Scoring job descriptions

The mistake that makes scoring undebuggable is doing both in one model call. When a score looks wrong you cannot tell whether the system misread the posting or misjudged the fit, and you end up tuning a prompt that was never the problem.

Run extraction first and store its output. Then score from the structured result. Now a bad score has a diagnosable cause, and improving scoring does not mean re-paying for extraction on every iteration.

The cost argument is as strong as the debugging one. Extraction is per posting and scoring is per candidate-posting pair, so a posting seen by four hundred candidates should be read once — and combining the two steps means reading it four hundred times.

Pull the posting into a comparable shape. The important distinction — and the one most implementations miss — is between what is genuinely required and what is listed hopefully.

Job descriptions routinely ask for more than the role needs, because they are written defensively. A system that treats every listed technology as mandatory will reject roles the candidate would comfortably get, which is the most common way these tools become useless.

Extract the requirement weight from how it is written rather than from where it sits on the page. "Must hold", a repetition through the responsibilities, and presence in the job title all signal a real requirement; a bulleted list of twelve technologies under a heading nobody wrote carefully does not.

• Hard requirements versus stated preferences versus aspiration
• Seniority, inferred from responsibilities rather than the title alone
• Work model and location, including contradictions between them
• Compensation if stated, with its currency and period
• Signals about the team: size, reporting line, stage

The naive comparison counts overlapping terms, which rewards padding and misses everything expressed differently. A candidate who has built data pipelines for six years does not match a posting that says "ETL" if you are matching strings.

Compare meaning instead, and score dimensions separately rather than producing one number: skills, seniority, domain, location and compensation each get their own judgement. A single blended figure hides the fact that a role is a perfect skills match in an impossible location.

Keep the gates out of the scoring entirely. Eligibility, a salary floor and a stated exclusion are checks that run before anything is scored, because the moment they become weights a confident skills match will eventually outvote them.

A bare score is useless to everyone. The candidate cannot tell whether to trust it, you cannot debug it, and it produces no signal for improving the system.

Require a short written justification naming the specific matches and the specific gaps. This costs a few tokens and changes the product: it lets a candidate overrule a judgement for a reason, and it lets you find out that the model has been marking down every role that says "fast-paced".

Require the reasoning to cite the posting rather than paraphrase it. A justification that quotes the requirement it matched against can be checked; one that describes the fit in general terms is a summary of the score rather than an account of it.

Scores drift for reasons that have nothing to do with fit: a model update, a prompt change, a reworded posting. A candidate who saw a role at 82 yesterday and 71 today has no way to interpret either number, and the credibility of the whole feature depends on that not happening casually.

Store the model version and prompt version with every score, and re-score deliberately rather than lazily. When the scoring changes materially, re-score the whole visible set at once so the candidate sees a consistent scale rather than a mixture of two.

Consider whether a numeric score is the right output at all. Bands — strong, worth considering, a stretch, not a fit — are less precise and more honest, and they do not invite a candidate to treat the gap between 74 and 77 as meaningful when it is noise.

A score is only meaningful relative to what the candidate would actually do. Build the labelled set early: thirty to fifty postings the candidate has marked apply or skip, with a sentence of reasoning where it is not obvious.

Measure agreement, and look at the disagreements individually — they are where the system is learning something. Often the model is right and the candidate was being inconsistent, which is itself worth surfacing rather than silently fitting to.

Keep that set as a regression test rather than using it once. Every prompt change, model upgrade and scoring tweak should be run against it before shipping, because the alternative is discovering a regression through a candidate who stopped receiving good roles a fortnight ago.