Knockout screening questions: 8 tests before automatic rejection

A knockout rule can reject a suitable applicant even when the software behaves exactly as configured.

Suppose the advert says “Zendesk or similar support software.” The form asks only, “Have you used Zendesk?” An experienced Intercom user answers no and is automatically rejected.

The eight tests below are designed to catch that kind of failure before real applicants are affected. They check the requirement, the evidence the system receives, how rules interact, where unclear answers go and what message the applicant receives.

Eight tests before automatic rejection

Use a test vacancy, fictional applications and contact details controlled by your team. Agree the expected result before submitting each case.

For every important rule, include a clear pass, a clear failure and an answer that needs clarification.

Turn each rule into a repeatable test

Before submitting anything, write down:

1. Requirement: what the job actually requires.
2. Accepted evidence: which answers or alternatives satisfy it.
3. Unclear case: which answer should go to a person.
4. Expected action: advance, reject or review.
5. Expected message: what the applicant should receive.

Then submit a fictional application and compare the actual result with that record. This makes a failed test diagnosable instead of merely surprising.

Submit through the same route real applicants use. A test path that bypasses the live rule does not tell you what applicants will experience.

Greenhouse-specific checks

• Use the normal job-post application route. Greenhouse documents application-rule exceptions for job post request emails in its application-rules guide.
• Test overlapping rules together. Auto-Reject takes precedence over Auto-Advance, and when two rejection rules match, rule order affects the result.
• Inspect rejected applications as well as the active queue. Greenhouse's auto-reject guide notes that auto-rejected applicants do not generate the usual new-application notification.
• Rerun the relevant cases after editing application questions or rules. Form changes can alter how an existing rule behaves.

Define the rejection rule before you automate it

A knockout screening question asks about an essential requirement where certain answers can remove an applicant from consideration.

Before automating that decision, agree four things with the hiring manager:

1. The exact requirement.
2. The evidence that can establish it.
3. The answers that clearly fail it.
4. The answers that need clarification or human review.

Agree the essential requirements before configuring the rule. Be precise about timing and acceptable alternatives: does the person need a qualification now or by their start date, and does the role require Zendesk itself or comparable support-software experience?

A working rule can still enforce an unsuitable requirement. In Singapore, MOM states that fair, merit-based employment expectations apply regardless of the technology used. In a 2023 US case, iTutorGroup agreed to pay $365,000 to settle an EEOC hiring-discrimination suit after the agency alleged that its application software used age thresholds to reject applicants.

Approve the requirement first. Then test whether the software applies it correctly.

Ask for the evidence the job actually needs

Suppose the role needs someone who has handled a support-ticket queue. If comparable platforms are acceptable, this question is narrow:

Have you used Zendesk?

This gives you more useful evidence:

Which support platform have you used, and what did you do in it? A few sentences are enough. If you haven't used one, say so.

A candidate can now describe the work they performed.

If the answer is only “Intercom”, you may still need a follow-up. An open question gives you more information, but it does not remove the need for judgement.

If the role genuinely requires Zendesk expertise from day one, ask about that work directly.

Test every accepted alternative

Suppose the role accepts either Zendesk or Intercom.

Two independent rules such as “reject if no Zendesk” and “reject if no Intercom” can accidentally require both.

Write the intended outcome before configuring the rules:

Experience with either accepted platform satisfies this requirement.

Then test Zendesk only, Intercom only, both and neither.

If comparable products are also acceptable, add an “other, please specify” route and decide who reviews those answers.

Treat missing data as missing data

A blank field and a “no” answer are different facts.

If the applicant supplied an answer and an import dropped it, recover or investigate the missing data before making a decision.

For Test 3, use the same sentence in two applications:

Managed a customer-support queue in Intercom for two years, including routing tickets and handling escalations.

Put it in the CV in one application and in a supported application-answer field in the other.

Keep the other relevant facts and all active rules the same.

Then check:

1. Did the answer arrive?
2. Was it available for assessment?
3. Did the application stay attached to the correct vacancy?
4. Did the correct rule act on it?

If the recommendation changes, investigate where the difference entered the workflow. The changed result does not tell you the cause.

If AI helps assess the answer, repeat an unchanged application too. AI output can vary even when the input does not.

After fixing a problem, submit a fresh application. A repair that only works on the example used to diagnose it is not enough.

If an unclear answer is supposed to go to a recruiter, confirm that the recruiter receives a visible task and can open the original answer. “Needs review” only works when somebody actually receives the work.

Mark any case you cannot safely reproduce as not tested, with the reason.

Test all active rules together

A rule that works alone can behave differently when other rules are active.

Test the combination that will actually run, then follow the application through the rest of the workflow.

Your ATS, screening tool and email service may control different steps. Correcting the application status in one system may leave a rejection email queued in another.

Inspect rejected applications as well as the active queue. Open the actual test rejection email, reply to it and check where that reply goes.

Test the consequence, not just the trigger.

Record failures so someone else can reproduce them

Save the test application, rule settings, expected result and observed result together.

For the support-software example:

Support role: alternative-software check

I submitted an application describing queue-management experience in Intercom. The job accepts Zendesk or similar experience, so I expected the application to remain under consideration.

Instead, it was rejected for “No Zendesk experience”, and a rejection email was queued for our test inbox.

We removed the product-specific rule and changed the question to ask about the work performed. On retesting, the answer reached the reviewer and no rejection was generated because of the software name.

[Link to test application, settings, date and person who checked it]

Keep the test application, rule settings and observed result together.

That result shows that the revised workflow sent the answer for review. It does not prove that automatic matching became more accurate.

Retest after changing the form

A form edit can change the logic that reaches an existing rule. Rerun the relevant cases after changing questions, answer options or rule settings, with all remaining rules active.

Ask for clarification only when it can change the decision

First check whether the answer already exists elsewhere in the application.

If your system lost information the applicant supplied, recover it rather than ask them to repeat it.

If the answer itself is ambiguous and the clarification could change the decision, ask a focused question.

For example, an applicant gives an expected salary of £55,000 for a role advertising £45,000 to £50,000 base salary plus bonus.

Before treating the answer as over budget, ask:

You mentioned an expectation of £55,000. Do you mean base salary, or total pay including bonus? Our advertised base range is £45,000 to £50,000, with bonus separate. You don't need to share your current salary.

Check the currency and pay period too.

Use the same precision for other requirements. Name the shifts that matter. State the date by which a qualification is required. Give the reviewer a concrete task such as “Ask Sam when they expect to qualify”.

If a candidate is waiting for your team to resolve an adjustment or support request, check any deadline that could expire while they wait.

For hiring in Great Britain, Acas's recruitment guidance covers discrimination protections and adjustment duties relevant to recruitment.

If a rule rejected the wrong applicants

Stop the rule and check any queued messages.

Then:

1. Save the earlier settings.
2. Identify every application assessed while those settings were active, including other vacancies using the same rule.
3. Search the affected answers as well as the stored rejection reason. Overlapping rules can hide some affected applications.
4. Reassess those applications against the actual job requirements using the information already supplied.
5. Assign someone to own the correction and the next applicant update.

Fixing the rule does not mean automatically advancing everyone it touched. It means applying the correct requirement again.

If the vacancy is still open and someone owns the review, a correction can be simple:

Subject: We're reviewing your support application

Hi Sam,

We made a mistake when reviewing your support application. Our screening rule required Zendesk experience, although we also accept experience with similar support software.

We're reviewing the decision using the information you already sent. I'll update you by [date], even if the review is still underway. You don't need to apply again or resend your CV.

I'm sorry for the mistake. Please reply if you have questions or no longer want to be considered.

[Your name]

Continue the review without requiring a reply.

Make the recovery path part of the rule design too. Name who investigates a challenged decision, make sure they can see the original answers and rule, and give applicants a way to reach them. For UK recruitment, the ICO's guidance calls for clear explanations and routes to challenge automated decisions or request human review.

Before enabling automatic rejection, follow three test applications all the way to the final message:

• a clear pass
• a clear failure
• an unclear answer

Record the expected and actual outcome for each one:

**Rule and vacancy:** [Name and settings link]
**Cases checked:** [Pass, fail, unclear and any accepted alternatives]
**Actual outcomes:** [Status, reason and message for each case]
**Unresolved cases:** [What still needs investigation]
**Until resolved:** [Keep automatic rejection off, or the approved narrower use]
**Who reviews exceptions:** [Name and visible work queue]
**Who can stop the rule:** [Name and contact]

Before switching the rule on, your team should be able to explain why an answer causes rejection, where unclear answers go and who can correct a mistake.