Your first AI screening pilot: 6 decisions before launch

A polished demo shows what the tool can do. A useful pilot shows which work your team can stop repeating and whether managers get better information.

Start your AI screening pilot with one vacancy, a defined group of applications and a named person responsible for each step. Agree what the tool should assess, what it may do automatically and what would make you pause it.

These six decisions turn those choices into a one-page pilot plan.

1. Which role has a problem worth solving?

Choose a vacancy with work you can describe and measure. Application volume matters, but so do the time spent reviewing each application and the questions repeated in first calls.

For the support-role example used here, recruiters repeatedly review similar CVs and ask applicants to explain how they investigated a customer problem. The team wants to reduce that repetition while keeping a person responsible for the next-stage decision.

Write the purpose in those terms:

Reduce repeated CV review and first-screen work for the Support Specialist vacancy, while preserving a recruiter’s review of the evidence before progression.

Before changing the process, time comparable CV reviews and first calls, including notes and follow-ups. Name the work you expect to stop doing. Compare that complete task with the new process, not with the time it takes an AI report to appear.

A specialist role with few applicants may have a different problem. If the delay is finding people or waiting for a hiring manager, screening automation may not be the first change to make.

2. What evidence will satisfy the requirements?

Have the hiring manager approve the requirements before configuring the screen. Separate essentials from preferences and record acceptable alternatives.

For the support role, the team needs evidence of investigating customer problems and providing a clear handover. Experience with a comparable ticketing system is acceptable; the company’s exact platform can be learned. A written customer response will be assessed in a later exercise rather than inferred from how confidently someone describes writing emails.

That gives the screen a defined task. It also tells the next interviewer what remains to be assessed.

Resolve unclear requirements before allowing them to exclude people. If the manager has not decided whether weekend availability is essential, keep that question with the manager rather than burying it inside the rubric.

3. What will happen automatically?

List the actual actions, not just “AI screening enabled”. Reading a CV, producing a report, sending an invitation and rejecting an application have different consequences.

For this pilot, the team chooses CV screening, manual interview invitations for selected applicants and supported result updates in the ATS. Recruiters retain the progression and rejection decisions. Candidate outcome messages remain with the existing recruiting process.

Our Stage Mapping and Candidate Updates guide, available to signed-in Screened users, separates those controls. Interview invitations are manual by default unless the team creates an automation rule; mapping an interview stage does not itself send an invitation.

Check the ATS’s own rules too. A tag or stage change can trigger work there even when you did not enable an equivalent action in Screened. Test the complete route with made-up applications and contact details controlled by your team before involving applicants.

Before using real candidate information, have the responsible hiring and privacy owners approve the data, recipients, candidate explanation and review process. A successful rehearsal shows that the tested steps work; it does not approve a new use of applicant data.

4. Who owns the work that remains?

Name the recruiter who reviews results, the manager who decides the next stage and the person who handles candidate questions. Give each a backup.

For the support pilot, the recruiter checks the reports during the working day and identifies unresolved questions. The hiring manager decides who should complete the customer exercise. The recruiting inbox receives candidate queries, with the recruiting lead covering absences.

Agree a response window the team can keep. The ability to finish an interview overnight does not mean a manager will read it overnight. Candidates need an honest explanation of the next step, not an expectation the employer cannot meet.

Also agree how reviewers can challenge an assessment. They need access to the source evidence and authority to change the decision, not just a button to acknowledge the recommendation.

5. What are the limits and stop rules?

Define the included applications before connecting a historical backlog. The pilot covers the next 60 applications within the agreed vacancy and intake period, with up to 20 interview invitations chosen by the recruiter.

Applications outside that group continue through the existing hiring process. Being outside the pilot is not a rejection; name who will keep that work moving.

Keep applications with unreadable files, missing results or other processing problems in the pilot log. They may need a different review route, but dropping them from the record would make the pilot look easier than the work your team actually received. Record any legitimate exclusions from the pilot and their reasons before interpreting the results.

Those are operating limits for this pilot, not a standard sample size or a promise that every product setting enforces them. Confirm which controls the system provides and how the owner will keep the live group within scope. A credit balance is not automatically a hard spending cap.

Agree a spending limit and a review date before any renewal or pilot opt-out deadline. Include setup fees and minimum commitments, not just the expected interview usage. Name the person who will check the bill and stop further spending when the limit is reached.

Decide what pauses the affected workflow: contacting the wrong candidate, attaching a result to the wrong job, exposing restricted information or misrepresenting an answer in a way that could change a hiring decision. Do not wait for a serious error to recur before containing it. Name who can act and what must be checked before resuming.

Rehearse the pause before launch. Check what happens to new intake, scheduled messages and interviews already underway, and confirm that the team can still access completed answers. An instruction to “switch it off” is not enough if nobody knows what that stops.

Keep candidate support and correction work moving during the pause, with an owner for each affected application.

6. What evidence would justify expansion?

Review the quality of the information, the candidate journey, human follow-through and the work removed. Do not expand merely because the team used the allowance.

For example, the pilot might reach 60 screened applications and 18 invitations. By the review date, 15 interviews are complete. Reviewers can prepare the next stage from 13 reports; two need clarification. Three invited applications remain incomplete, including one reported interruption.

Those observations give the team specific work to inspect. Why were the two reports unclear? What happened to the three unfinished interviews? Are their deadlines still open, and have their requests been answered? Which questions did the human interviewers stop repeating?

Review the CV stage as well as completed interviews. Compare some low-rated or uncertain assessments with the source and the recruiter's decision, including relevant experience described in unfamiliar terms. These pilot-quality checks supplement the individual decision reviews agreed in the plan; they do not replace them.

Compare the staff time with the baseline from decision 1. Include reading all reports, the two clarifications and help with unfinished interviews. Keeping every original first call means you have not saved that call time. Faster CV review or shorter calls might still produce an overall saving, but measure those changes rather than assuming them. If the pilot adds work overall, explain what better information justifies it.

The counts do not establish predictive accuracy or a general time-saving rate. Use the findings to decide what to fix, what to stop and whether another comparable group is justified.

One-page AI screening pilot plan

Use the six decisions to complete one working plan. This support-role example still needs your team’s names, dates and spending limit:

Before signing off the plan, replace job labels with the actual vacancy, name the owners and backups, and enter the intake dates, spending limit and review date. Keep the baseline timings beside it so the final comparison starts with the work you agreed to change.

Record material changes during the pilot. If a requirement or question needs correction, fix it and note which applications were assessed before and after the change. Do not combine different configurations into a single reassuring performance number.

For the support pilot, the review must explain the two unclear reports, account for the unfinished interviews and show whether repeated staff work fell. Then record the next decision: stop, fix and repeat, or extend to another bounded group.

Carry forward what worked, not an assumption that every vacancy needs the same interview. Approve the next role’s requirements, owners and candidate route before increasing the workload.