What we tested
The AI screened and ranked a deliberately mixed set of technical applicants. Scenarios included conventional senior engineers, career changers, returners, candidates with adjacent experience, incomplete CVs and superficially strong profiles with weak underlying evidence.
Human practitioner panel
Each practitioner scored scenarios independently before seeing the other reviewers’ answers. That lets us distinguish genuine consensus from group discussion.
Priority findings
F01
Strong unconventional candidates were consistently under-ranked
The system performed well when a candidate’s experience closely matched the wording of the role, but the practitioner panel recovered several candidates whose transferable experience was stronger than their titles or CV vocabulary suggested.
F02
Confident ranking persisted when evidence was incomplete
When CV information was sparse or ambiguous, the AI still produced a precise ranking rather than signalling that a recruiter should verify a key assumption.
F03
Keyword-aligned experience received too much weight
Profiles that mirrored the job description language could outrank candidates with deeper but differently expressed experience. Practitioners described the issue as “surface relevance beating actual evidence.”
F04
Explanations described the ranking but did not always justify it
The product usually produced readable explanations, but reviewers wanted clearer links between specific evidence, uncertainties and the final recommendation.
Recommended product changes
When required evidence is missing or conflicting, show what needs verification rather than forcing a confident rank.
Add career changers, returners, adjacent-domain candidates and differently worded equivalent experience to the permanent eval suite.
Require recommendations to point to the specific CV evidence that drove the rank and explicitly state unresolved assumptions.
Retest plan
After the fixes, rerun the same 12 scenarios plus four newly held-out edge cases. The release would be considered materially improved if:
Without reducing performance on conventional profiles.
Practitioners agree the AI knows when not to make the final call.
At least 8/10 practitioner usefulness across held-out scenarios.
Methodology snapshot
Independent first pass
Practitioners evaluate scenarios separately to avoid anchoring and groupthink. We capture decision, confidence, trust, rationale and what they would do next.
AI-assisted synthesis, human evidence preserved
AI helps cluster patterns, quantify disagreement and draft findings. Practitioner judgements remain attributable in the evidence layer rather than being replaced by an AI summary.
Want this applied to your AI workflow?
The founding pilot covers one important workflow, three matched human practitioners, structured scenarios and an executive report for £950.