AI interview candidate experience checklist: test before launch

Use this AI interview candidate experience checklist before launch to test the invitation, interview flow, recovery steps and hiring-manager report from beginning to end.

Ask a colleague to complete the interview without coaching, then ask a hiring manager to use the result. A controlled rehearsal can expose unclear instructions, unhelpful questions and support problems before applicants encounter them. Assessment accuracy and fairness still need separate checks.

Choose one role and one controlled application

Use an authorised test vacancy with made-up application content and contact details the tester controls. Confirm the phone number as carefully as the email address. A fictional name attached to a real stranger’s number is not a safe test.

Give the application enough detail to support a meaningful conversation. For a training-coordinator role, the person might have arranged workshops, reconciled attendance records and followed up missing feedback. Include one clear example and one genuine gap for the interview to explore.

Do not upload a folder of previous candidates or a customer’s records because they are convenient. The first rehearsal needs controlled inputs, not representative hiring data. Broader evaluation comes later through an approved process.

A job named “Test” is not automatically isolated. Ask the workspace and ATS administrator which settings apply to it and whether creating or connecting it could import existing applicants or publish a vacancy.

Screened’s Stage Mapping guide, available to signed-in users, explains that some integration settings apply across connected jobs. An ordinary recruiter should not change those shared settings without the responsible administrator.

Check what can happen automatically

Before adding the test application, inspect the relevant intake, interview-invitation, stage-movement and candidate-message rules in both systems. Identify any action that could affect a real person.

Also inspect work already queued. Turning off a future rule is not proof that a previously scheduled action has disappeared.

For example, Teamtailor’s trigger guide describes delayed actions and shows scheduled triggers under Upcoming in the candidate’s Activity. It provides a separate cancellation action. Check the equivalent mechanism in the ATS you actually use rather than assuming every product behaves the same way.

Agree who can stop the test if something unexpected happens. Have the administrator available for settings the recruiter cannot change. The aim is a small controlled exercise, not an unsupervised experiment on a live hiring campaign.

Check usage charges too. Screened’s credit guide distinguishes completed interviews from pending reservations. Confirm the terms for your account; a trial allowance does not make every subsequent internal test free.

Begin at the message the candidate receives

Send one invitation through the intended workflow and inspect the inbox. Did it arrive once? Does it name the employer and vacancy? Does the recipient understand what the interview involves and where to ask for help? Compare it with the browser and phone invitation templates.

Follow the current invitation link. Do not replace this step with a marketing demo that has already filled in the candidate’s details and bypassed the employer’s instructions.

The UK Government’s usability-testing guidance recommends observing people attempting specific tasks to see whether they understand and can complete them. Apply that principle to your colleague: give them the task, then watch where they hesitate instead of explaining every screen in advance.

Ask them to describe uncertainty while reading instructions or choosing a next step. During the interview itself, let them answer normally; narrating the test into the microphone would change the conversation you are trying to inspect.

Record the confusion in the person’s words. “I thought this was a video interview and went to find a different room” gives you a useful invitation change. “Improve candidate experience” does not.

Give the interviewer something worth clarifying

For the training-coordinator test, use this made-up answer:

We had three attendance lists that did not agree. I compared the registrations with the facilitator’s sign-in sheet and found that two people had switched sessions. I updated the attendance record and asked the facilitator to confirm the remaining missing names.

A useful conversation can explore the applicant’s checks, the source they relied on and what remained unresolved. It should not credit the applicant with designing a company-wide attendance system or assume that every discrepancy was their own mistake.

Then try a request for clarification: “Do you mean how I checked the lists, or how I told the trainer about the changes?” Listen for whether the response helps the candidate answer the intended question.

Finally, ask something outside the approved company information:

Would the manager let me work from home on Fridays?

The interviewer should explain the confirmed working arrangement and direct the question to the hiring team if it cannot answer it. It should not invent a benefit. Before testing, agree what the employer has authorised it to say about this role.

Follow that contact route during the rehearsal. Can the tester reach the named recruiter and get an acknowledgement with a response date? Finding the question in a transcript is not the same as assigning someone to answer it. Do not promise automatic forwarding unless the actual workflow does it.

Screened’s interview guide describes role questions, follow-ups and reports for human review. Your rehearsal checks whether the configured interview uses those capabilities usefully for this vacancy.

Ask a hiring manager to use the report

After processing finishes, give the hiring manager their normal access and a concrete task: decide what they learned from the attendance example and prepare the next interview question.

Do not explain the answer before they open the report. You need to find out whether the report itself makes the applicant’s contribution clear.

Can the manager find the relevant answer? Does the summary distinguish checking records from obtaining a facilitator’s confirmation? Can they tell what remains unknown? Does the next stage need to repeat the first interview, or can it move to a useful new exercise?

Inspect the ATS result as well as the source report. Confirm the job context, date and destination. A correct assessment hidden behind an inaccessible link is not yet a working handover.

Check when the result should become available under the documented process. If it does not arrive as expected, investigate before assuming that the test failed or sending another invitation.

Test a dropped browser interview or a phone reschedule

For a browser interview, a controlled interruption can show what the candidate sees and what the recruiter receives. For a scheduled phone route, check the booking time and confirmation, then reschedule the test appointment and check what happens to the original booking.

Use the supported instructions. Screened’s troubleshooting guide distinguishes an available Try again action from cases needing follow-up or support. Do not assume that every interrupted conversation resumes exactly where it stopped.

Check the retained answers and status. Topics never reached should not become weak answers in a completed assessment. Confirm which attempt a reviewer should use after recovery.

Stop the exercise if it begins to contact a real candidate or affects another vacancy. Preserve the relevant operational record and involve the responsible administrator before trying again.

Record what passed and what needs fixing

Suppose your training-coordinator rehearsal produces these observations. The record should show the problems that still prevent an invitation to real candidates:

Do not average these into a passing score because three rows look good. Duplicate instructions and an unusable help route still need fixing before the wider invitation.

Give each failed row a specific correction and repeat the affected journey. If removing an ATS message changes the stage behaviour, recheck that behaviour too. A fix can have consequences beyond the screen where it was made.

Clean up without triggering another hiring action

Before removing the test data, check pending interviews, scheduled messages and downstream actions. Archiving is not always a neutral cleanup step.

Screened’s incomplete-interview guide warns that archiving can move or disqualify an ATS application and that restoring in Screened does not reverse that action. Use the authorised cleanup procedure and confirm the final state in both systems.

Keep the short test record and necessary evidence of any issue, not an unnecessary archive of recordings and personal details. Use the applicable retention policy for the controlled test material.

What this rehearsal lets you conclude

A successful rehearsal shows that your colleague could complete this interview and your manager could use its report. It supports trying the tested setup in a small, monitored pilot.

It is not proof that real candidates will have the same experience, that the assessment measures job ability accurately or that excluded applicants were correctly excluded. Plan those evaluations separately, with appropriate data and review.

The first objective is simpler: do not ask hundreds of applicants to be the first people who discover how your process actually works.