AI screening for Ashby: test the Screened integration
Test Screened’s Ashby integration before buying. Check job-specific reports, recruiter and manager access, corrected assessments and missing results.
Adan Vielma · · 8 min read
An applicant looks suitable for customer support but has not described managing a team. They apply for both support and team lead. After the team-lead assessment, their support application disappears from your strong-match list.
The support assessment may not have changed. Screened keeps a result for each job, but the score returned to Ashby sits on the person’s shared profile. A later assessment can replace the score your list uses.
Test how your team uses the result, not just whether the systems connect. Check job-specific reports, manager access, corrections and missing results.
Use made-up applications and contact details your team controls. Before connecting a test job, have your administrator check shared settings and existing applications that could be imported. Our ATS configuration guide explains those controls (Screened sign-in required).
Where do Screened’s results appear in Ashby?
Screened can add fields, tags and notes to the candidate's Ashby profile, plus a link to their record in Screened.
Open the application with your usual account, not an administrator's demo account. Check that you can read the assessment and open the CV it describes. Start with these two fields:
AI Screened CV Results contains the written assessment and its reasons. Read that first and check the most important claim against the CV.
AI Screened CV Fit (%) contains the score. Ask what it measures and what happens when the CV leaves a requirement unanswered. Do not read it as the probability that the person will succeed in the job. Our Ashby results guide lists the returned fields, notes and filters (Screened sign-in required).
Ask the vendor to show how you start an assessment, check progress and stop new submissions. Which actions happen in Ashby, and which require Screened or support? A returned score tells you none of that.
Build the list your recruiter will open each morning, then compare it with the applications sent for screening. Include people still waiting for a result or a readable CV. A list that shows only scored applicants can hide precisely the cases that need attention.
Complete one review and record the next step. Reopen the list: can a colleague tell that you have reviewed the application, rather than merely received an AI result? Check that the action you used has not also sent a message or moved the application unexpectedly.
What happens when the same candidate applies for two jobs?
One person can apply for several jobs. Their candidate profile brings those applications together, but each job application still needs its own hiring decision.
Screened keeps each job's assessment separately. Ashby displays the returned fields, tags and Screened link on the person's shared profile. A later assessment for another job can therefore replace what you see in those fields.
Use one test candidate with two job applications:
- Create a test CV describing investigation of billing complaints and customer updates. A supervisor approved refunds. Use it for a support application, screen that application and open AI Screened CV Results.
- Confirm that the assessment describes the support role. Save the support review list and note what the shared fields display before testing the second application.
- Add the same test candidate to a team-lead role requiring management experience absent from the CV, then screen that application. After it finishes, return to the support application and saved list. Check both which role the visible result describes and whether the support application still appears.
Both assessments may remain available in Screened even when Ashby displays the wrong one for the application you are reviewing. The test needs to check what your saved list hides as well as what it shows.
How do you avoid reviewing the wrong job's score?
Until you have tested the score filter, use the job's applications and your team's review notes to find people awaiting review. Do not rely on the shared score alone to decide who appears. Include applications still waiting for a report.
Open the assessment under the intended job in Screened, then record the decision on the corresponding Ashby application. A note could say: “Reviewed the support assessment and CV. The team-lead score on the shared profile was not used.” Add the actual recommendation and its reason.
This approach uses the job-specific report; it does not create a new job-specific score field in Ashby. Confirm that your saved list keeps the application visible after the second assessment.
Count the extra work. Opening the right report may be manageable. Regularly rebuilding the review list because suitable applicants disappear is a problem to resolve before rollout.
Can the hiring manager use the report to prepare an interview?
Ask the manager to open the support application with their own account and prepare one follow-up question. Let them find the evidence without the vendor directing every click.
A short handover could say:
The applicant describes investigating billing complaints and keeping customers updated. Their supervisor approved refunds. Ask them to describe one complaint: what did they find, and what did they ask their supervisor to do?
Check that the manager can open the original CV or interview answer when needed. If not, fix access or use another approved way to share it. Don't make the candidate repeat the whole first interview because a link didn't work.
If interviewers should assess candidates independently, check whether earlier recommendations are visible in notes and fields. Hiding a formal scorecard may not hide a copied recommendation elsewhere.
Explain why a score changed
Did the candidate send a new CV? Did the manager change a requirement? Or did the original report misread something? Write down the reason. “Score updated” does not tell the next recruiter which of those happened.
Stay with the support test application and add this sentence to a new CV version:
I compared the items on the bill with the customer's order and found that one item had been charged twice. I recommended a correction, and my supervisor approved the refund.
Use the supported reassessment process. The new sentence explains what the applicant found. It still says the supervisor approved the refund.
Record that the applicant supplied new detail. A better result after a fuller CV is not, by itself, evidence that the original screen was wrong.
Reopen the link saved with the first decision. Check whether it still shows the original evidence or now points only to the replacement. Ask how to retain the history needed to explain each decision before relying on that link.
Check that the manager sees a corrected claim
Return to the support CV's statement that a supervisor approved refunds. Ask the vendor how you would handle a report that wrongly gives that authority to the applicant. Follow the supported correction or support process; don't assume a recruiter can edit the AI's assessment directly.
After correction, ask a colleague to open the application and tell you who approved the refund. Let them find the answer. If they rely on an old brief, correct that copy too; fixing the source report has not yet fixed what the interviewer reads.
Then check whether the old result still affects a score, filter or automatic action, and have the responsible colleague fix that too. Do not base a hiring decision on the disputed claim while it is being checked.
If a misleading email already went out, arrange an explanation for the candidate. Editing the report cannot undo a delivered message.
Why is the screening report missing from Ashby?
Screened's Ashby fields, tags and notes may not appear at exactly the same time. A written assessment can arrive as a note while a field is still empty.
Before retrying, work out what has already happened:
| What you see | What to check |
|---|---|
| The candidate hasn't appeared in Screened. | The connection, added job and stage selected for importing candidates. |
| The candidate is in Screened, but there is no assessment. | The CV, any processing error, whether screening is active and available credits. |
| The assessment is in Screened but not where expected in Ashby. | The connection, result settings and candidate notes. |
| The assessment names another job. | The candidate's separate job assessments in Screened. |
Read the error before trying again. AI Screened Error - Duplicate means you should check the earlier assessment for that job. Missing Resume calls for a usable CV, while Limit Reached calls for available screening credits. Resending the application won't fix a missing file or depleted credits.
If a note is missing, ask your administrator to check Candidate Updates → Notes to post. Missing fields may have a different cause. Contact support before reconnecting an integration used by live jobs; a reconnect can affect more than this one application.
Give support the job and candidate links, then describe what worked and what's missing:
The CV report is ready in Screened, but I cannot find it in Ashby. Please check the return for this application.
**Job and application:** [Links]
**Completed report:** [Screened link]
**Completed at:** [Date, time and time zone]
**Already checked:** [Fields, notes and any displayed error]
Fill in the details before sending. Keep the original application and activity available while support investigates. Recreating the profile or repeatedly resending it may make the sequence harder to reconstruct.
What should an Ashby screening integration prove before you buy?
Match each test application to its report or an unresolved problem. A matching total isn't enough: three reports could include two for one application and none for another.
Summarise what your team completed and where it needed help. A trial update could read:
Recruiters and managers could open the reports with normal access. For people applying to two jobs, we needed to open the job-specific assessment in Screened. We still need a reliable way to build the support-job review list before using score-based filters.
Separate setup from recurring work. Giving a manager access once is different from repairing a review list every time a candidate applies for another job.
An extra click can be worthwhile for a useful report. An application disappearing from review needs fixing. Before buying, make sure your colleagues can find the right assessment and take the next step without the installer beside them.
