The Shortlist Problem: why good CVs aren’t enough

You send six promising CVs to the hiring manager and get this reply:

“Good backgrounds, but I was hoping for someone more senior.”

Do they mean someone who has handled larger accounts, led difficult negotiations or managed a team? Until you know, another afternoon of reviewing applications may produce different names and the same feedback.

A shortlist needs more than good CVs. It should give the manager a reason to meet each person and a useful starting point for the interview.

For each person you recommend, answer three questions: Why meet them? What is still unclear? What should we ask next?

Turn “more senior” into a clear requirement

Ask the manager: “What should this person be able to handle without you taking over?”

For a store supervisor, that might mean running a busy shift rather than having worked on one. For an account manager, it might mean leading a contract renewal rather than helping someone else prepare.

Suppose an account-management team keeps hearing about customer problems just before contracts expire. The new hire needs to spot the warning signs earlier, understand the concerns and involve colleagues while there is still time to respond.

That is a clearer brief than “own relationships and grow revenue.” Agree the limits, too: the person may lead negotiations but still need approval for a discount. Working independently does not mean deciding everything alone.

Geoff Smart and Randy Street make a useful distinction in Who: The A Method for Hiring. Their role scorecard separates why the job exists, what the person must accomplish and the abilities and behaviours needed to do it.

Agree what the team can teach

For this vacancy, experience negotiating renewals might matter more than knowing your software. Ask the manager: “Would we rather teach our software to someone who has handled difficult renewals, or teach renewals to someone who knows our software?”

The answer can be “we need both.” If you agree that something is teachable, check who will teach it and whether they have the time.

Now compare three fictional CVs:

• Eight years at a well-known company, supporting renewals for large accounts. The setting looks relevant, but the responsibility is unclear. Did they negotiate terms or prepare material for someone else?
• Three years at a smaller company, negotiating renewals within agreed pricing limits. The CV describes relevant responsibility. Check whether the accounts presented similar challenges to those in this role.
• Customer-support experience, with no detail about renewals. Handling complaints may be useful preparation. Ask about contract negotiations before deciding whether that is a skill the person would need to learn.

The second CV describes the relevant experience more clearly. That alone does not make the applicant a better fit. Clarify important gaps in the other promising applications before deciding who to interview.

If the discussion changes a requirement, revisit all affected applications. Do not make an exception only for the candidate who prompted it.

Do not ask the CV to answer every hiring question

A CV that says “managed contract renewals” gives you relevant experience to explore. It may not explain how the person handled a difficult price request or decided when to seek approval. Those are useful interview questions, not necessarily reasons to reject the CV.

A skill can be essential to the job without being something a CV can establish. Agree what experience justifies a conversation, then decide what that conversation needs to test.

Separate missing information from missing experience

“Supported contract renewals” should not become “led renewal negotiations” in a recruiter’s notes or an AI summary. But it should not become “has never negotiated a renewal,” either.

When a recommendation looks wrong, identify the problem before changing a requirement or score.

The CV is unclear. Ask: “What were you responsible for in those renewals?” A focused clarification may be enough when one missing detail could change the next step. Use the same approach for other applicants with the same gap.

The experience is related, but not the same. Resolving complaints is not the same as negotiating contract terms. Consider whether it is sufficient preparation given the training available. If negotiation experience is essential from day one, do not count related work as a confirmed match.

The applicant says they have not done the work. “I have never negotiated a renewal; I passed updates between customers and account managers” is a clear answer, not missing information. Apply the agreed requirement, including any comparable experience the team has agreed to accept.

The review missed a detail. The CV describes leading renewal negotiations, but the assessment says that experience was not mentioned. Check the passage against the exact requirement. If it supports a match, correct your review, reconsider the candidate’s next step and investigate the error. Keep a valid requirement unchanged when the problem is how the CV was read.

You do not need to chase every omission. Focus on details that could change the next step for an otherwise plausible applicant. Then check the remaining requirements. Resolving one question does not settle the whole assessment.

Ask for detail, not agreement

A follow-up should help a candidate explain their work, not supply a story for them to agree with. Consider this fictional exchange:

Interviewer: “Tell me about a time you improved a process.”

Candidate: “We had a backlog. I worked with the team and we got it under control.”

Interviewer: “So you worked out what was causing the backlog, divided up the work and tracked the results?”

Candidate: “Yes.”

The candidate has agreed, but most of the detail came from the interviewer.

A better follow-up to the original answer would be:

Interviewer: “What was your part in that?”

Candidate: “I noticed the same questions kept coming up, so I drafted answers the team could reuse.”

Interviewer: “How did you decide which questions to start with?”

Now you have a decision to explore. Ask whether colleagues used the answers and how the candidate checked the results. If the candidate worked through assigned cases instead, ask about that work. Doing the work well and changing the process are different contributions; which matters depends on the role.

Decide how you will judge the answer

For the renewal role, agree what you need to understand: how the applicant spotted a risk, chose a response and worked within their authority. Then ask: “Tell me about a renewal you thought might be lost.”

Ask what happened and what else affected the result. A customer may have stayed because a product problem was fixed. The applicant might have helped resolve that problem, or had little part in it. Ask what they did rather than judging their negotiation skills from the outcome alone.

If you have not established the result, say so in the notes. If the person says “we,” ask what they did and what others did. “My manager approved the change; I wrote the answers and trained the team” tells you more than “I transformed the process.”

The US Office of Personnel Management’s guidance on structured interviews calls for a set sequence of questions and consistent scoring criteria across candidates. Agree how you will judge the answers before hearing them.

Put the next interview question in the shortlist

“Strong candidate—thoughts?” leaves the manager to reconstruct your review.

Instead, explain why the person is worth meeting and what remains to be explored. After an initial screening call for the fictional account-manager role, the note might read:

Why meet them: In the screening call, the applicant described leading renewal negotiations and identifying a product problem early enough to involve colleagues before the contract expired. That is relevant to our need to act earlier on renewal risks.

What is still unclear: They said price changes needed approval. We have not yet asked how they handled a customer request they could not approve.

What to ask next: “Tell me about a renewal where a customer asked for terms you could not approve. How did you handle it?”

A question you have not asked is not a weakness you have found. If you already asked and the applicant could not give a relevant example, record that instead.

If you are shortlisting from CVs alone, use the same format and identify the CV as the source. Keep the original material accessible, and distinguish what the applicant told you from anything you checked independently. Use these follow-ups alongside the core questions and standards for all candidates.

A manager may reasonably recheck an important answer. If they repeat the whole first screen, ask what detail was missing or why they did not trust the report. Keep any conversation you still need, and count its time when judging how much work the process saves.

Using AI? Review more than the top matches

When AI helps build the shortlist, apply the same standard to its recommendations. A convincing set of matches does not tell you who might have been missed.

For an initial check, use applications you are authorised to assess. Include clear matches, plausible applicants with negative recommendations and unclear CVs. Ask the manager to record who they would advance and why before seeing the AI results.

Compare the reasons for disagreements. Did the tool overlook a sentence? Did the manager assume responsibilities from a familiar company name? Are you interpreting the requirement differently?

Return to the CV and the agreed criteria rather than assuming either judgment is right. Have a person review disputed decisions. If a CV could not be processed, resolve that separately rather than treating the missing assessment as a rejection.

Use this small review to find problems, not to estimate the system’s accuracy or fairness across the whole applicant pool. It cannot tell you how many suitable candidates were missed overall.

For AI interviews, read the questions as well as the report. Check that it describes what the candidate said, without attributing ideas from a leading question to them. A natural-sounding conversation is not proof of an accurate assessment.

Look beyond completed interviews

Before inviting candidates, explain the task, how AI is involved and how their answers will be used. Say who will review them, what happens next and where to get help. Test the invitation and help route, and check that the team can provide any promised human review or alternative.

Then look at who could not finish, asked for help, requested a conversation or did not respond. A microphone problem, a question about data use and a preference for speaking to a recruiter need different responses. None, on its own, tells you whether someone can do the job. Silence does not explain why someone stopped.

Where another route is offered, assess the same job-related abilities without making the alternative unnecessarily longer or harder.

Keep the human conversations that help candidates decide whether they want the job. Collecting their work history does not replace explaining the opportunity and answering their questions. For some roles, CV screening alone may be enough.

Measure the work left for the team

Compare two ways of reaching a useful shortlist from the same applicant pool. The tasks do not have to be identical: removing repetitive reading is part of the point. But count the human checks the new process actually needs.

For illustration, reading 600 applications at 90 seconds each takes 15 hours. Suppose AI-assisted review takes six hours, with another three for setup and corrections. That saves six hours at this stage, not fifteen. These are example inputs, not measured Screened results.

Now count the remaining calls, clarifications and manager review. If those steps take longer than before, they reduce the saving. Keep one-off setup separate from recurring work, but include both. If the new process takes more time overall, faster reports have not saved the team work.

Saving review time does not necessarily mean hiring sooner. If twelve candidates are waiting and the manager can review four this week, thirty more reports may not get interviews moving any faster. Agree the next batch and when the manager will respond. A full calendar is a scheduling problem, not evidence that an applicant is unsuitable.

If you are paying for a tool, compare the benefit with the cost of the CV screening and interviews you actually need. Before investing time in a trial, check what is included, extra usage charges, billing frequency, any minimum term and what you pay between hiring rounds. Time available for other work can be valuable without reducing payroll or other spending.

Before you reopen the applications

Turn “someone more senior” into a requirement you and the manager can both explain. For the renewal role, that might be: “We need someone who has negotiated renewals. We can teach our software. The interview should explore how they handle a customer request they cannot approve.”

Then revisit the affected applicants. The next shortlist should explain why each person is worth meeting and what you still need to learn—not simply replace one set of promising CVs with another.

To try this on a current role, start with Screened’s CV screening. Set the agreed requirements, then check the reasons behind the recommendations.