In an earlier posting we saw how ranking resumes can save time spent by recruiters and hiring managers in the recruitment process. We also saw that it lends itself well to lean hiring by enabling selection of small batch sizes.
Experiment – Manually Ranking Resumes
We developed a game for ranking resumes by comparing pairs with some reward for the winner. The game didn’t find the level of acceptance we expected it to find. So we enlisted a human expert to do the ranking. It took half a day for an experienced recruiter to rank 35 resumes. Very often the recruiter asked which attribute was to be given higher weightage? Was it experience, location, communication or compensation?
These questions indicate that every time we judge a candidate by his resume, we assign some weightage to various profile attributes such as experience, expected compensation, possible start date, etc. Every job opening has its own set of weightages, which are implicitly assigned as we try to compare the attributes of a resume with the requirements of the job opening.
So the resume-ranking problem essentially is reduced to finding the weightages for each of the attributes.
Challenge – Training Set for standard ranking algorithms
Many algorithms are involved to solve the ranking problem. Most of the ranking algorithms fall under the class of “Supervised Learning” which would need a training set consisting of resumes graded by an expert. As we saw earlier, this task is quite difficult as the grade will not only depend on the candidate profile but also on the job requirements. Moreover, we can’t afford the luxury of a human expert training the algorithm for every job opening. We have to use data that is easily available without additional efforts. We do have some data from every job opening as hiring managers screen resumes and select people for interviews. It’s easy to extract this data from any ATS (Applicant Tracking System). Hence we decided to use “Logistic Regression” that predicts the probability of a candidate being shortlisted based on the available data.
We have seen that “Logistic Regression” forecasts the probability based on weightages for various attributes learned from which resumes were shortlisted or rejected in the past. This probability – in our case – would indicate if the candidate is suitable or not. We would use this number to rank candidates in descending order of suitability.
In our company, we had access to data of the following 13 attributes for approximately 3000 candidates who were screened for approximately 100 openings over the last 6 months.
Earliest Start Date
We needed to quantify some of these attributes such as education, stability, communication, etc. We applied our own judgment and converted the textual data to numbers.
We were unsure if we would get consistent results as we were falling short of historical data of resumes. We ignored openings that had 10 or less resumes screened. On the other hand, we also discovered a problem with large training sets – particularly in the case of job openings that remain open for long. These job openings are likely to have had a change of requirements. As we learned later, consistent accuracy was obtained for job openings having training sets whose population was in the range of 40 to 80 resumes.
Running Logistic Regression
We had listed 22 openings for which several hundred resumes were presented to the hiring managers in the last 6 months. We maintained a record of interviews scheduled based on suitability of the resumes. We decided to use 75% of the available data to train (Training Set) and 25% to test (Test Set) our model. The program was written to produce the following output:
Vector of weightages for each one of the 13 attributes
Prediction whether the set of test cases would be “Suitable” or “Unsuitable”
The result was based on the accuracy of the prediction. Accuracy is defined as the following formula:
Accuracy = (True Positives + True Negatives) / (Total # of resumes in the Test Set)
“True Positives” is the number of suitable resumes correctly predicted to be suitable. Similarly “True Negatives” are the number of unsuitable resumes predicted to be unsuitable. We achieved average accuracy of 80% ranging from 67% to 95%.
Efforts to improve accuracy
A Plot of suitable and unsuitable resumes on Experience vs Pay didn’t show any consistent pattern. The suitable resumes tended to be more of highly paid individuals who had lower experience, which is somewhat counterintuitive. Other than this, the suitable resumes tended to cluster closer to the center of the graph as compared to the unsuitable ones.
Given the nature of the plot, the decision boundary would be non linear – probably a quadratic or higher degree polynomial. We decided to test using a 6th degree polynomial – thus creating 28 attributes from 2 main attributes – viz. experience and pay. We ran the program again; this time with these 28 sixth degree polynomial and remaining 11 attributes thus a total of 39 attributes. This improved the accuracy from 80% to 88%. We achieved 100% accuracy for 4 job openings.
Regularization had no impact on accuracy. Hence we didn’t use any cross validation set for testing various values of the regularization parameter.
Values of weightages or parameters varied slightly each time we ran the program to find the minimum of the cost function. This indicates that the model found a new minimum in the same vicinity every time we ran the program with no changes to the training set data.
Taking a close look at the chart above, we observe the following:
One job opening gives extremely negative weightage to “Current Compensation” – this means that higher paid candidates are not suitable. Interestingly, the opposite is the case for most other job openings.
C++ Developer position assigns positive weightage to “Total Experience” but negative weightage to “Relevant Experience”. The requirement was for a broader skillset beyond just C++.
We can continue verifying the reasons for what turns out to be a fairly distributed set of values for weightages for various attributes. Each job opening has pretty much independent assessment of the resumes and candidates.
As expected we observed that accuracy generally increases with sample size or size of the training set. As previously mentioned, accuracy was reduced in the case of job openings that remained open for long periods of time and the selection criteria underwent change.
“Synerzip team is very responsive & quick to adopt new technologies. Team naturally follows best practices, does peer reviews and delivers quality output, thus exceeding client expectations.”
“Synerzip’s agile processes & daily scrums were very valuable, made communication & time zone issues work out successfully.”
“Synerzip’s flexible and responsible team grew to be an extension to the StepOne team. Typical concerns of time zone issues did not exist with Synerzip team.”
“Synerzip worked in perfect textbook Agile fashion – releasing working demos every two weeks. Though aggressive schedules, Synerzip was able to deliver a working product in 90 days, which helped Zimbra stand by their commitment to their customers.”
“Outstanding product delivery and exceptional project management, comes from DNA of Synerzip.”
“Studer product has practically taken a 180% turn from what it was, before Synerzip came in. Synerzip cost is very reasonable as compared to the work they do.”
“Synerzip makes the timezone differences work FOR the customer, enabling a positive experience for us. ‘Seeing is believing’, so we decided to give it a shot and the project was very successful.”
“The Synerzip team seamlessly integrates with our team. We started seeing results within the first sprint. And due to the team’s responsiveness, we were able to get our product to the sales cycle within 7 months.”
“Product management team from Synerzip is exceptional and has a clear understanding of Studer’s needs. Synerzip team gives consistent performance and never misses a deadline.”
“Synerzip is different because of the quality of their leadership, efficient team and clearly set methodologies. Studer gets high level of confidence from Synerzip along with significant cost advantage of almost 50%”
“Synerzip’s hiring approach and practices are worth applauding. Working with Synerzip is like
“What you see is what you get”.”
“Synerzip has dedicated experts for every area. Synerzip helped Tangoe save a lot of cost, still giving a very high quality product.”
“Synerzip gives tremendous cost advantage in terms of hiring and growing the team to be productive verses a readymade team. Synerzip is one company that delivers “co –development” to the core!”
“Synerzip is a great company to work with. Good leadership and a warm, welcoming attitude of the team are additional plus points.”
“Our relationship with Synerzip is very collaborative, and they are our true partners as our values match with theirs.”
“Synerzip has proven to be a great software product co-development partner. It is a leader because of its great culture, its history, and its employee retention policies. ExamSoft’s clients are happy with the product, and that’s how ExamSoft measures that all is going well.”
“They possess a great technical acumen with a burning desire to solve problems. The team always takes the initiative and ownership in all the processes they follow. Synerzip has played a vital role in our scaling up and was a perfect partner in cost, efficiency, and schedules.”
“As we are a startup, things change on a weekly basis, but Synerzip team has been flexible in adapting the same”
“Synerzip team has been very proactive in building the best quality software, bringing in best practices, and cutting edge innovation for our company.”
“We’ve been working for more than six years with Synerzip and its one of the better, if not the best, experience I’ve had working with an outsourcing company.”
“My experience with Synerzip is that they have the talent. You throw a problem at them, and someone from that team helps to solve the issue.”
“The breadth and depth of technical abilities that Synerzip brings on the table and the UX work done by them for this project exceeded my expectations!”
“Synerzip UX designers very closely represent their counterparts in the US in terms of their practice, how they tackle problems, and how they evangelize the value of UX.”
“Synerzip team understood the requirements well and documented them to make sure they understood them rightly.”
“Synerzip is definitely not a typical offshore company. Synerzip team is incredibly communicative, agile, and delivers on its commitments.”
“Working with Synerzip helped us accelerate our roadmap in ways we never thought possible!”
“While working with Synerzip, I get a feeling of working with a huge community of resources, who can jump in with the skills as needed.”