Measuring the Quality of Those You Didn’t Hire

Are You Missing the Best?

by Dr. John Sullivan

The quality of those not hired is the most valuable recruiting metric that you have never heard of! It informs you how often your organizations is failing to hire the highest quality applicants.

A few years back I was advising a Fortune 100 firm that had a painfully slow and somewhat arrogant hiring process. To demonstrate the negative impact of their process I had to prove to a skeptical senior manager that they were letting top candidates get away. I asked a manager hiring for an important job to rank, in order of quality, 100 applicants who had been sourced for the role. The chosen rank was discretely written on the back of paper copies of the candidate's resumes. Months after the role had been filled, the manager was asked if they were satisfied with the hire. He was, and felt quite certain that he had successfully hired a "top 5" candidate. After hearing of his satisfaction I had him look at the initial rank he had provided the candidate who was later hired: 75.

You can imagine his shock when he realized that the hiring process had somehow let every single one of the top-ranked applicants that the firm had prided itself in hiring "every single time" slip away. Clearly the quality of the people who they didn't hire was significantly higher than the quality of the one that they did.

 

Selecting HR Metrics Is Unfortunately Not a Scientific Process

Most organizations adopt metrics based on those covered by benchmark reports or that can be easily enabled via their technology providers, instead of determining what they need to discover or prove. As a result, many organizations are burdened with data and reports that offer little in the way of guidance helping them improve their effectiveness. One metric often not fully taken advantage of is quality of hire, which I estimate less than 40% of organizations even attempt to use. Even fewer use the quality of hire derivative, quality of those not hired, because it can very quickly demonstrate how poorly a process performs.

 

 

Determine Where Your Recruiting Problems Are Occurring

During an advisory conversation with a recruiting leader at a well-known social networking firm experiencing difficulty achieving hire diversity, I asked "at what step or stage is your recruiting process failing?" I wasn't surprised when he responded "we don't actually know, we just know that the overall recruiting process is not producing the results we need." Like many organizations, this organization lacked well-thought-out metrics that enable both performance reporting and process diagnostics.

Recruiting processes fail because either they do not attract enough top-quality candidates up front, or they fail to accurately identify, assess, and sell those attracted on the job at later stages in the process. Most organizations focus heavily on measuring sourcing effectiveness, but ignore the later stages of the process altogether. One benefit of using a "quality of those not hired" metric is that it focuses exclusively on the back end, where I estimate at least 50% of those organizations not meeting their goals have problems. If you doubt that the problem is post-attraction, ask your favorite agency or executive recruiter what percentage of qualified candidates are lost due to slow or ineffective actions on the part of hiring managers and corporate recruiting processes.

One of the purposes of the quality-of-those-not-hired metric is to force organizations with a high percentage of quality hires slipping away to identify where in their process the talent opts out or gets dropped. There are six post application stages where firms lose top candidates, including:

 

 

1. Resume screening process — the ATS, a recruiter, or a hiring manager mistakenly screens out top applicants.
2. Telephone screen — top applicants rank poorly on their phone screens or their screen cannot be completed, so they are dropped from consideration.
3. Interview scheduling — they get frustrated over the number of interviews and dropout or they cannot complete them in time because of scheduling conflicts.
4. Interview assessment — they voluntarily drop out before the interviews can be completed, or the interview process mistakenly rates them poorly.
5. The offer process — either the process fails to include most of the top applicants on the list of finalists, or they reject the offer.
6. Reference checking — even though they are high-quality candidates, they somehow fail the reference/background check.

 

Since the goal of a good metric is to help you identify what is not working, carefully select and implement at least one metric that can point out failures occurring during the latter stages of your recruiting process.

 

 

Focus Only On the Top Candidates

A quality-of-those-not-hired metric can become cumbersome if it attempts to categorize the quality of every applicant who doesn't get hired. In order to save time and money, narrow your focus to the strategic issue of "what happened to the cream of the crop?" Out of 50 applicants for a single job, there might only be three who were so qualified that a hiring manager would actually regret failing to hire them. I call these individuals "regrettable misses," and it is these folks that the quality-of-those-not-hired metric aims to highlight.

 

 

Action Steps for Developing a "Quality-of-Those-Not-Hired" Metric

If you decide to implement a quality-of-those-not-hired metric, there are several action steps to consider, including:

 

 

• Setting goals — I recommend that you set goals for the use of this metric that include: accurately identifying the top three to five "regrettable" candidates; determining what percentage of top candidates become finalists for the position, and determining what percentage of new hires came from the top candidate list.
• Select an evaluation range — this metric should focus solely on reporting the progress of "the very top applicants" who senior managers would regret not hiring. To limit the scope of evaluation, preselect what size of candidate slate will be evaluated. For most jobs, three to five top applicants would be a sufficient number to track. You can also use a set percentage of all applicants (i.e. top 10%) to define what you mean by top.
• Determine when to identify top applicants — identify the top applicants early on in the hiring process so that you will have time to address any issues that emerge before a final hiring decision is made. If you are conducting an audit post hire, you need to make sure that the person doing the initial selection isn't aware of which individuals were finalists and who was hired.
• Select methods for identifying top applicants — the best method for identifying the top applicants is to have multiple evaluators select a finalist slate that is then merged to create the sample that will be monitored. An alternative approach involves using the profile matching capabilities of your ATS to produce a listing of top applicants. A third possible list segments applicants who come from high-value benchmark firms.
• Report the metric in percentages — the best way to report the quality of "those not hired" metric is in percentages. For example: 66% of all finalists came from the top-ranked list, and 47% of the time a top-five-ranked candidate was hired.
• Identify the stage where top talent slips through — for high priority and mission-critical jobs, after the hiring process is complete, identify at what specific stage in the recruiting process did a top applicant opt out or get dropped from consideration. You can then use that information to improve that stage.
• Identify cause for top candidate removal from consideration — if a significant number of top candidates opt out or are dropped from consideration without becoming finalists, follow up and find out why. If your process screened them out prematurely, recruiters and hiring managers must be questioned to identify what knockout criteria is being applied. If the candidate dropped out on their own, they need to be questioned to see if their early withdrawal could've been prevented.
• Keep in touch — separate from the process of calculating the metric, the organization should keep in touch with and build a relationship with the high-quality applicants who you regret missing. Building this relationship will help to ensure that they will favorably consider another opportunity with your firm in the future. Develop an alert system so that the star applicants can automatically receive e-mail alerts whenever a relevant job opens up.

 

 

 

Sample "Quality of Those Not Hired" Report

Here is a sample report illustrating what a recruiting leader or hiring manager might see.

Job Family: ASIC Engineer

Report Period: Q3 2010

Hire Volume: 53

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Metric	Job Family	Organization
Percentage of top candidates in finalist pool	53%	52%
Party responsible for removal from consideration
-candidate filtered out by recruiting process	17%	48%
-candidate opted out of recruiting process	83%	52%
Percentage of top candidates who rejected offer	95%	22%
Percentage of hires from top candidate slate	31%	42%
Process stage contributing largest slate loss	Interview Scheduling

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Recommended Actions

Candidates reported that more often than not interviews would need to be rescheduled because times initially proposed were no longer available upon confirmation. Many candidates reported that it took recruiting coordinators longer than a week to confirm meeting dates and times. Solution:

 

 

1. Allocate dedicated time slots to recruiting activities that cannot be booked by other activities more than 24 hours in advance.
2. Establish service level agreements that call for manager response to scheduling inquiries within four business hours.

 

 

 

Final Thoughts

If you were a competitive fisherman participating in a pro fishing tournament and you repeatedly landed prize-winning fish, you would be justifiably proud. However, if you repeatedly caught prize contenders but lost them prior to tournament completion, wouldn't you want to know exactly where and why you kept losing them? That is exactly what the "quality-of-those-not-hired" metric tells you. It reports how often you successfully land a great applicant, but fail to convert them to employee. Your organization can't attain the highest level of new hire on-the-job performance (quality of hire) if your process allows the highest-quality applicants to be missed.

 

 

About the author

Dr. John Sullivan is a well-known thought leader in HR. He is a frequent speaker and advisor to Fortune 500 and Silicon Valley firms. Formerly the chief talent officer for Agilent Technologies (the 43,000-employee HP spin-off), he is now a professor of management at San Francisco State University. He was called the "Michael Jordan of Hiring" by Fast Company magazine. More recruiting articles by Dr. Sullivan can be found in the ER Daily archives. Information about his numerous other articles, books and manuals about recruiting and HR can be found online.

Sullivan, John. "Measuring the Quality of Those You Didn't Hire — Are You Missing the Best?" ERE.net. 26 July 2010. Web. 29 July 2010.