Evaluating a matching model

Every matching system reaches a moment when someone asks whether the new approach is better, and the honest answer is that nobody knows. Building the evaluation afterwards is possible and always biased by what you have already shipped.

A few hundred CV-and-posting pairs, judged by people who understand hiring, is enough to be decisive. Get judgements on a scale rather than a binary: "strong fit", "worth applying", "wrong seniority", "not eligible" distinguish failures that a yes/no label collapses.

Sample the awkward cases deliberately rather than taking a uniform draw. Career changers, non-standard backgrounds, senior candidates and roles with unusual titles are where matching actually fails, and a random sample of typical pairs mostly confirms that easy cases are easy.

Accuracy is the wrong frame. Nobody consumes a matching system as a classifier — they look at a ranked list and read the top of it. So measure the top: whether the good matches are in the first handful, and how deep you must go to find them.

Recall at a realistic cut-off tells you whether the right jobs are reachable at all. A rank-weighted quality metric tells you whether they are near the top. Both are needed: high recall with bad ordering feels broken to a user who never scrolls.

Measure retrieval and ranking separately. A good role that never reached the shortlist cannot be recovered by a better ranker, and the two failures are indistinguishable in a single end-to-end number while needing entirely different fixes.

• Recall at the depth users actually read, not at 100
• A rank-weighted quality score for ordering
• Eligibility violations counted separately — these are bugs, not misses
• Coverage: the share of postings that are ever recommended to anyone

Offline evaluation measures agreement with labels. It cannot measure whether people apply, whether employers respond, or whether the feed feels stale after a week. Those only appear online.

So use offline evaluation as a gate — it catches regressions cheaply and fast — and online experiments as the decision. Anything that passes offline and then fails online has taught you something about your labels, which is worth recording.

Choose the online metric carefully, because clicks reward the wrong thing. A feed optimised for click-through converges on well-known companies and attractive titles, which produces engagement rather than employment — so the metric that decides should be applications that received a response.

Matching systems learn from historical patterns, and hiring history contains patterns nobody wants to reproduce. An aggregate metric that improves can hide a subgroup for which results got worse.

Slice the evaluation: by years of experience, by career-break presence, by non-standard background, by region. Look at whether the score distribution differs for equivalent candidates. This is a legal exposure as well as an ethical one, and aggregate numbers are specifically unable to reveal it.

Test with matched profiles differing in one signal as well as slicing real traffic. Slices find harm that already happened; matched pairs find the mechanism, and running both is what separates a discovered problem from an explained one.

Agree in advance what a failed fairness check triggers. A threshold with no defined consequence becomes a number somebody explains away under delivery pressure, and the time to decide that a material disparity blocks a release is before one appears.

The dangerous failures are not the ones that throw errors. A model version changes, an extraction step improves, a chunking parameter is tuned, and the scores shift for reasons unrelated to fit. Everything still returns results, so nothing alerts.

Keep a fixed regression set with expected outcomes and run it in CI on every pipeline change. It is unglamorous and it is the single highest-value piece of evaluation infrastructure most teams are missing.

Include the components upstream of the model. An extraction regression, a chunking change or an approximate index quietly losing recall all present as worse matching, and a suite that only exercises the final score cannot tell you which of them moved.

A labelled set degrades. Postings close, the market shifts, the roles you are matching against stop resembling the ones you labelled eighteen months ago, and a suite that only ever goes up is usually measuring its own age.

Refresh part of it on a schedule and keep the rest fixed. The fixed portion catches regressions; the refreshed portion keeps the evaluation representative, and mixing both avoids the two failure modes of a stale benchmark and a moving target.

Guard against tuning to the set. A system that improves on your labels for six months while users report no difference has been fitted to the evaluation rather than to the problem, which is exactly what a holdout group exists to detect.