Designing a relevance score

Most match scores mean nothing in particular. They are a weighted blend of whatever signals were available, scaled so the output looks like a percentage. Users read them as a probability, which they are not.

Pick a definition and hold to it. "The share of stated requirements this candidate demonstrably meets" is defensible and checkable. "The estimated probability of reaching an interview" is also defensible, and needs outcome data to back it. What you cannot do is leave it undefined and still expect trust.

Consider whether a number is the right output at all. Bands — strong, worth considering, a stretch, not a fit — carry the same decision with none of the false precision, and they stop a candidate reading meaning into the gap between 74 and 77.

Requirements are not interchangeable. A hard eligibility constraint, a must-have skill and a nice-to-have preference behave completely differently, and blending them into one average destroys the distinction.

Treat eligibility as a gate: fail it and there is no score, because no amount of skill match makes an ineligible application worth submitting. Then score must-haves and nice-to-haves with separate weights, and report them separately.

Weight by how the posting stated the requirement rather than by where it sat. A skill in the job title, repeated through the responsibilities, is the real requirement; one item in a bulleted list of twelve rarely is, whatever heading was above it.

• Eligibility — binary, gates the whole score
• Must-have requirements — heavily weighted, individually reported
• Preferred requirements — lightly weighted
• Contextual signals — seniority band, domain adjacency, recency of the skill

A score can rank candidates correctly and still be useless if the numbers do not mean what they appear to. If everything lands between 70 and 90, the user cannot distinguish a strong fit from a weak one, and the score stops informing any decision.

Check the distribution on real data. Scores should spread across the range, and if you claim a probability, roughly the stated fraction of the 70% cases should actually succeed. Most scoring systems have never been checked this way.

The compression usually comes from averaging. Blending six partly correlated signals pulls everything towards the middle by construction, so a clustered distribution is a modelling result rather than a statement that all candidates are similar.

A bare percentage invites the one question you must be able to answer: why. Without a breakdown, a user who disagrees with the score has no way to tell whether the system missed something in their CV or whether they genuinely lack the requirement.

Show which requirements were met, which were not, and what evidence was used. This also turns the score into something useful rather than merely informative — an unmet requirement the candidate can actually address.

Phrase an unmet requirement as what the CV shows rather than what the person lacks. About half of flagged gaps in practice are things the candidate has and never wrote down, and "your CV does not evidence this" converts directly into a stronger application where "you lack this" sends them away.

Scores drift. Postings change in style, your extraction improves, a model version changes, and the same candidate scores differently for reasons unrelated to fit. Nobody notices because there is no alarm on a number that looks plausible.

Keep a fixed regression set of CV and posting pairs with expected scores, and run it whenever anything in the pipeline changes. It is the cheapest safeguard available and almost nobody builds it.

Store the model and prompt version with every score, and re-score a whole visible set at once when the scoring changes materially. A candidate comparing a role scored yesterday against one scored today is comparing two scales, and that is worse than either scale alone.