Using two AI screening tools in Ashby: what should the second tool add?

Using two AI screening tools in Ashby only makes sense if the second tool adds a distinct piece of useful work.

Decide what you need it to add before paying for another opinion. It might replace the first review, check difficult cases or interview candidates to collect missing information. Test the workflow you actually intend to run, including how recruiters will resolve disagreements between the two systems.

Do you need another CV review or a follow-up question?

Read the rest of the application before adding another assessment.

For the applicant whose contribution is still unclear, ask:

Your CV mentions supporting 45 hires. Could you describe your part in one of those searches? Which stages did you handle, and which did a colleague handle? One example is enough.

A reply such as “I sourced and assessed candidates; my manager negotiated the offer” gives you something concrete to explore. Ask how they assessed applicants and who chose the shortlist. If this job also requires offer negotiation, that remains a separate question.

Check what Ashby’s result actually measures

Before buying another assessment, open the reasoning behind the relevant criterion in Ashby’s Feedback tab. Its AI-Assisted Application Review guide distinguishes meets, does not meet and undecided. An undecided result may correctly reflect the missing detail in “supported 45 hires”; another confident score would not supply that detail.

Check the column you are comparing too. Average Score comes from human application-review ratings. AI Job Criteria Met Percentage shows the proportion of AI criteria met. An empty human score does not mean the AI evaluation failed, and these measures should not be averaged with a second supplier’s fit score.

Ashby’s application-form check also asks whether a question was answered, not whether the answer demonstrates the required skill. “Answered” and “qualified” are different findings. Identify the information missing from your current review before deciding what another tool needs to add.

Compare both tools using the same application

Start with applications your team is permitted to use for the comparison. Ask your administrator to keep the test separate from live hiring actions. Check the scope of shared settings before changing automatic messages or stage moves.

Have a colleague read the CV and recommend a next step before seeing either score. Compare their reasoning with both reports afterwards. A disagreement starts an investigation; it does not automatically establish that the person or tool is right.

Give both tools the same CV, application answers and requirements. Check profile data too: Ashby’s guide says its AI can consider education and work history on the candidate profile when there is no résumé. A difference in available information is not proof that one tool reads the same evidence better. Keep the tools’ recommendations out of each other's inputs, using an isolated test setup rather than deleting live notes.

Check why an evaluation was skipped before sending the application elsewhere. Ashby surfaces AI opt-outs separately; do not treat an opt-out as a processing failure or assume it permits routing the person into a different AI tool. Follow the candidate’s recorded choice and the reviewed data-use process.

Choose applications that answer the question you are testing. If you want the second tool to find people the first screen missed, include excluded applications. Running it only on Ashby’s shortlist cannot show whether it recovers anyone outside that list.

Assess an added interview separately. It may collect information neither tool could find in the CV. That is useful, but it is not proof of better CV reading. Compare the interview with the email or call you would otherwise use.

If you adjust the screening instructions, test fresh applications too. A fix that handles the familiar example may still fail on a different one.

Can two screening tools agree and still miss a good applicant?

Both tools can miss the same passage or enforce the same unintended requirement. Review applications both would decline, not only the disagreements.

Keep unreadable CVs and missing reports in the test too. Find out whether those applications receive a proper review or disappear because there is no score to compare. Agreement between completed reports says nothing about that missing work.

For applications with clear recommendations, compare these four groups:

Match the remedy to the cause. Correct a report that missed the sourcing example on page two. Fix an instruction requiring offer negotiation when the brief did not. Ask the applicant when the CV leaves their responsibility unanswered.

The more confident report is not necessarily the better one. Its explanation must fit the source and the agreed job requirement.

Use difficult cases to investigate failures, then a normal mix to estimate everyday work. A second tool may help with unclear CVs but add little to obvious cases.

Selective use can make sense. Count the work of selecting, though: reading every CV to decide which needs another review may use up the time you hoped to save.

What should you do when two screening tools disagree?

If both tools must approve an applicant, either can block them. One mistaken rejection can outweigh a correct recommendation. Assign someone to examine disagreements before adding that second hurdle.

For the comparison trial, keep decisions with your team. A disagreement rule could read:

The recruiter will check both explanations against the CV and agreed requirements before recommending the next step. Disagreement alone will not trigger a rejection.

Reserve time for that review. At six minutes each, 30 disagreements take three staff hours. Use your own timings in the plan. Record how many checks corrected a recommendation and how many merely reconciled wording.

Automatically advancing anyone recommended by either tool has the opposite risk: unsupported recommendations may pass through. Treat disagreement as a reason to inspect the evidence, not a decision in itself.

How an extra rejection step can reduce the interview group

Start with 100 applications your human review considers ready for the same next stage. The first tool advances 90. The second advances 80% of those 90, which is 72.

The first tool stops 10 applications and the second stops another 18. Inspect those 28 decisions against the same requirement. A strong shortlist of 72 does not establish that the exclusions were justified.

Here, 80% means 72 of the 90 applications reaching the second tool advanced. It is not an overall accuracy figure. You cannot substitute two vendors' advertised accuracy rates for these figures.

A later task that assesses a different skill can legitimately rule someone out. That is different from two tools judging the same CV against the same requirement.

Don’t average away the disagreement. One tool may accept everyday software use while the other expects system configuration. An average gives you a number, not a decision about which skill the job needs.

Record the explanation and the hiring action separately. Replacing the first assessment, using the second only for difficult cases and requiring both to approve everyone are three different processes. Test the one you intend to run.

What did the second assessment add?

Ask what you learned from the second assessment that the first did not provide. Did it recover overlooked experience, collect a missing answer, introduce an error or just rephrase the same information?

A trial can show that the second tool found experience the first missed. It cannot, by itself, show that the people it recommends will perform better after hiring. That is a different evaluation question; the OPM assessment glossary explains the distinction.

For the recruiter CV, record findings like these:

Spend review time on differences that change the decision. An invented offer-negotiation responsibility matters. Two accurate summaries phrased differently may not.

A useful conclusion could be: “The second report found the sourcing example on page two. We still need to ask who chose the shortlist.” That tells the recruiter what to keep and what to do next.

Show the manager what the extra step taught you

For the recruiter applicant, give the manager the clarified responsibilities and a question to explore:

Please ask how this applicant assessed candidates in the search they described and who chose the final shortlist.

They say they sourced and assessed candidates. Their manager negotiated the offer, so we still need to check the applicant's own negotiation experience if the role requires it.

The manager can now examine how the applicant assessed candidates instead of collecting their career history again.

Our guide to reviewing Screened results in Ashby also explains how to use Screened after an existing screening stage (Screened sign-in required). Test the combined process before enabling automatic moves. Check interview invitations separately.

Choose the next change from what you found. Fix a bad instruction in the first tool rather than pay another supplier to compensate for it. Use a second review where it reliably uncovers missed evidence. Add an interview when you need to ask the candidate something the CV does not answer.

Before buying, name what improved and what it cost in money and review time. Two agreeing scores are not enough. A useful answer to “what did this person actually do?” is a much better starting point.