Getting better, honestly

A system that learns what a user engages with and shows more of it looks like it is improving: engagement rises, dismissals fall. What has actually happened is that the range collapsed.

Real improvement means surfacing things the user would not have found and would value. That is the harder thing to measure and the only one worth claiming.

Tracking the two separately is what keeps you honest. Relevance and diversity can both be measured week on week, and a system where relevance climbs while diversity falls is narrowing — which is a diagnosis you can only make if you were recording both.

Three areas improve reliably with use. Preferences that were explicitly corrected are simply better data. Form and site handling learned once can be replayed, which makes applications faster and cheaper. And knowledge of which employers respond accumulates usefully.

Note what these have in common: each is a concrete fact learned and stored, not a statistical inference from thin behavioural data.

They also improve at different rates and for different reasons. Form handling improves per site and plateaus; corrected preferences improve fastest in the first fortnight; employer response data improves across all users at once, which is the only one of the three that benefits a new user on their first day.

• Corrected preferences — direct, unambiguous, high value
• Learned site and form structures — replayable, cheap
• Employer response behaviour — accumulates across users
• Documents that were confirmed and sent — a growing evidence base

Predicting success from one person’s outcomes is not achievable at the sample sizes a job search produces. Neither is inferring why an application failed, or learning what phrasing works from a handful of responses.

Systems claiming these are typically reading noise. The responsible position is to say the data does not support the claim, rather than to produce an insight because the interface has a space for one.

The design pressure here is real and worth naming. A dashboard with an empty insights panel looks broken, so somebody fills it, and the filling is where an honest product quietly becomes a dishonest one. "Not enough data yet" is a legitimate thing for a panel to say.

A job search changes over its course. What someone wanted in month one may not be what they want in month four, and an agent weighting all history equally is anchored to a person who no longer exists.

Decay behavioural signals, keep explicit statements longer, and let a new explicit statement override everything before it. Adapting to the current person is the whole point of adapting at all.

Watch for the moments that should reset rather than decay. A changed target, a new constraint or a long gap in activity all suggest the accumulated model is describing a previous search, and continuing to average across the discontinuity produces a feed that serves neither version of the person.

Nearly every available metric moves in the wrong direction under narrowing, which is why this is hard to check casually. Click-through rate, dismissal rate and session length all improve as a feed converges on a comfortable band.

The measurement that survives is a holdout. A small proportion of users served the unpersonalised ranking, compared over months on applications and responses rather than clicks, is the only way to know whether personalisation did anything — and it is cheap compared with being wrong about it for a year.

Ask users directly as well, occasionally and specifically. "Did this feed show you anything you would not have found yourself?" is a better question about improvement than any behavioural proxy, and it is the one the product is actually claiming to answer.

• A permanent unpersonalised holdout, however small
• Diversity tracked alongside relevance, always both
• Applications and responses as the outcome, not clicks
• One direct question, asked rarely

A system silently adjusting its behaviour is one the user cannot correct or trust. When the agent learns something that alters what it does, say so in a line.

And provide a reset. Sometimes the model of a person is simply wrong — a strange week, a search on someone else’s behalf — and the fastest fix is to start again rather than to argue with accumulated inference.

Make the reset granular where you can. Clearing learned preferences while keeping application history and form knowledge is usually what someone wants, and an all-or-nothing reset means they either keep a broken model or lose the record of their own search.