LLM re-ranking for job listings

A model cannot look at fifty thousand postings. It can look at fifty, and it will order those fifty better than a similarity score will, because it understands that six years of one framework implies competence in a related one.

So the model sits last. Cheap retrieval produces the candidate set, filters remove the ineligible, and the model orders what remains. Anyone trying to use a model as the primary search is solving the wrong problem expensively.

That placement also caps the damage it can do. A re-ranker cannot recover a good role that retrieval never surfaced, and it cannot admit one that a filter excluded — which is worth knowing when a bad result appears and you are deciding which layer to investigate.

Scoring each posting independently is parallelisable, cheap to cache and stable — the same pair always gets the same treatment. Its weakness is that independent scores cluster: without seeing alternatives, the model rates most things a 7.

Judging a whole list at once produces better relative ordering, because comparison is easier than absolute judgement. It costs more per call, cannot be cached per posting, and introduces position bias — the model favours whatever appeared first. Shuffle the order across runs if you go this way.

Listwise also has a context ceiling. Twenty full postings and a CV is a large prompt, so the practical form is short summaries in the list with a separate fetch for detail — which is the same discipline that keeps any agent context usable.

• Pointwise: cacheable, parallel, stable — but poorly spread
• Listwise: better ordering — but position-biased and uncacheable
• A practical middle: pointwise to shortlist, listwise on the top ten

Handing the model a CV and fifty postings and asking for a ranking gets you the model’s own idea of what matters, which will not be the candidate’s. It tends to over-weight title similarity and under-weight things the candidate cares about, like compensation or the seniority step.

State the criteria explicitly and in priority order. Ranking quality improves markedly, and — more importantly — it becomes a thing you can tune rather than a black box you argue with.

Keep the hard constraints out of it entirely. Eligibility, location and a salary floor are filters that ran before this step, and re-introducing them as criteria invites the model to trade one off against a strong skills match.

Require a one-line reason per result. It costs very little, it is what makes a recommendation credible to the candidate, and a reason that reads as thin is a reliable signal that the match was thin.

Re-ranking on every page load, for every user, is the fastest way to a bill nobody approved. The cost is per comparison, so it scales with traffic rather than with catalogue size, which is the direction that hurts.

Cache aggressively against a key of the profile version and the posting version. Re-rank only the first page of results, since almost nobody reaches the third. And set a hard budget per request that degrades to the cheaper ordering rather than failing.

Precompute the feed rather than re-ranking interactively where you can. A daily ordering computed in batch and served from a lookup costs a fraction of the same work done per page view, and the underlying profile changes slowly enough that it rarely matters.

Track cost per user per day, not only the total. One user whose configuration triggers forty times the average is invisible in an aggregate, obvious per user, and always a configuration problem rather than a pricing one.

The same pair can score differently across runs, and a prompt change can reorder everything. That is tolerable in a ranking and intolerable if you show the score as a stable number to the user.

Keep a fixed evaluation set with known-good orderings and run it on every prompt or model change. Without that, "the ranking got worse" is a report you cannot act on.

Store the model and prompt version with any score you persist, and re-rank a whole visible set at once when the ranking changes materially. A candidate comparing a role scored yesterday with one scored today is comparing two scales.

Consider not showing the number at all. Order is what the candidate consumes, and a position in a list carries the useful information without inviting anyone to treat a difference of three points as meaningful.