Signal vs Noise: Reference Checks

Over the next few posts, I want to dive into hiring.
How do you design a good interview process? How do you source great candidates? And ultimately, how do you separate signal from noise well enough to make a good hiring decision?

Let's start near the end: reference checks.

I often hear founders tell me that reference checks aren't useful. “Candidates only give me references who will say nice things about them.” Of course they do. The mistake is expecting the reference to do the evaluation for you.

My hot take is I would rather skip the interview rather than the reference process. Interviews can be flawed and biased in so many ways. Plus some candidates just don’t interview well (I definitely don't consider interviewing a skill of mine). The best way to judge a candidate is to understand their previous work, accomplishments, and relationships. Then ask whether the environment that produced those successes can be replicated at your company. References can get you all of that information, the trick is asking the right questions.

Everyone knows candidates are going to give you references who like them. That doesn’t invalidate references or their importance, it’s just a constraint you need to design around.

The right questions

How do you get someone who may be a diehard friend of your candidate to be honest about their true strengths and weaknesses? It's really simpler than you may think. 

1. Ask for concrete examples and stories

2. Don't demand criticism; ask questions that make honesty socially easy.

Ask for evidence, not opinions.

Let’s say one of the concerns you have about your candidate is their ability to act like a leader. A bad question to ask one of their references is, “Is this person a good leader?” Because that’s extremely easy for a reference (who's trying to help their buddy) to say back, “Of course! I think they’re an excellent leader.” If instead you ask, “Can you tell me three stories of the candidate acting like a leader when you all worked together? What was the context and what was happening in the business at the time?” If leadership is a quality your candidate has, it should be easy for the reference to tell lots of stories of their leadership. If instead, they struggle to come up with examples that may be a signal that this is a trait they still need to work on. One person struggling to come up with examples may be that person’s poor memory. But if every reference you talk to struggles to come up with examples, then you know that's a real gap you should be aware of. Additionally, when asking examples, always remember to ask follow-up questions from examples they give. My favorite follow-up to ask is, “What was the impact the candidate had due to their actions in that example?”

Make honesty easy

The other way to create signal for yourself is phrasing your questions in such a way that you give the reference permission to be honest. Don’t ask binary questions that force the reference to either lie but be a good friend or be honest but feel like a crummy friend. Instead ask questions in such a way that they can continue to be a good friend while being honest. 

For example, ask questions asking the reference to place the candidate in percentiles. As in, “In your company was this person bottom 50% at what they do? Top 50%? Top 25? Top 10? Top 5 or 1%?” If the reference enthusiastically says “top 25%,” that sounds like praise in isolation. But if you're hiring for a role where you're specifically looking for the exceptional few, the fact that they didn't say top 5 or 1% may itself be useful signal.

Or ask questions asking the reference to take themselves out of the equation. For example, “What would this candidate's biggest detractors say about them? Why do they feel that way?” You’re not asking the reference to say negative things. You’re just asking them to quote what other people are saying. Or ask them a neutral question that seems to have nothing to do with performance like, “What kind of environment does the candidate need to perform their best?”

Think about the questions you're asking and in what position you're putting the reference in. Don’t force them to betray their friend. Instead word your questions in such a way that they get to preserve their loyalty and be honest at the same time.

Wrapping up

The best way to know if someone is going to perform well is to learn about their previous performance. Some folks are better at tests or interviews than others. But previous performance is much harder to game than an interview. This is why references are so important. Even just the number of references someone can provide can be a signal. When hiring executives at Lattice, one signal we paid close attention to was how easily a candidate could produce references. Someone who could immediately give us a list of 20+ former managers, peers, and reports who were happy to talk was telling us something before we even made the first call.

If you’re not getting the signal you need out of your reference process, don’t drop the step. Instead think about how you can reengineer the process and the questions you ask to get you the signal you need.