AI observability, explained

Conventional monitoring watches for failure: error rates, latency, saturation. An LLM system that has quietly become useless produces none of those signals. It returns 200s, in normal time, with well-formed text that happens to be wrong.

This is the defining problem. There is no exception to catch, because nothing threw. The system is not broken in any sense your infrastructure understands, and the first report will come from a user who noticed something a dashboard cannot see.

The consequence is that observability here has to measure the content of responses rather than the fact of them. That is a different discipline from availability monitoring, which is why bolting it onto an existing stack rarely works without deliberate effort.

The unit worth recording is the whole request, not the model call. When an answer is wrong, the question is which input caused it — and that is unanswerable unless you kept the assembled context rather than the template you think you rendered.

Teams that log the final rendered prompt find bugs in an afternoon that they had spent weeks attributing to model quality. It is the single highest-return thing to add.

Record what retrieval discarded as well as what it returned. Knowing that the correct passage was ranked eighth when you only passed five is a specific, fixable diagnosis; knowing only that the answer was wrong is not.

• The final assembled prompt, not the template
• What retrieval returned, with scores, including what was discarded
• Every tool called, its arguments and its result
• Model, version and parameters at the time of the call
• Token counts in and out, and end-to-end latency
• The user-visible output, so a complaint can be traced to a trace

Everything above involves storing what users typed and what the system retrieved on their behalf, which in most products means personal data and sometimes a great deal of it. Useful observability and careless data retention are very easy to build at the same time.

Decide the policy before the logging: what is retained, for how long, who can read it, and whether sensitive fields are redacted at write time rather than at query time. Redacting on the way in is the only version that survives someone exporting a dataset later.

Sampling helps here as well as with cost. Full-fidelity traces for a percentage of traffic, with metadata-only records for the rest, gives most of the diagnostic value at a fraction of the retention exposure.

Latency and error rate still matter, but they are the least interesting numbers. The ones that tell you whether the system is healthy are quality and cost, and neither appears in a standard observability stack.

Tail latency deserves particular attention. Generation time varies with output length, so an average hides the requests that took eleven seconds. Users experience the tail, not the mean.

Refusal rate is the most diagnostic single number and the one most often missing. A system that quietly becomes more willing to guess improves on every other metric while getting worse in exactly the way users notice.

• Pass rate on a fixed evaluation set, tracked over time
• Cost per request, and which changes moved it
• Refusal and fallback rate — a sudden rise means something upstream broke
• Retrieval hit rate: how often the needed passage was even fetched
• p95 and p99 latency, not the average

Automated checks catch what you thought to check for. Everything else is found by people reading real outputs, which means a deliberate sample — a fixed number per day, reviewed on a schedule, with disagreements written down.

Bias the sample towards the interesting: low-confidence answers, unusually long conversations, sessions where the user rephrased the same question repeatedly. A user asking the same thing three different ways is the clearest unlabelled signal of failure you will get.

Feed what you find back into the evaluation set. A failure discovered by a human once and never encoded as a test case will be rediscovered by a user later, which makes the review expensive rather than valuable.

The dependency at the centre of your system is one you do not version. A provider can change model behaviour with no deployment on your side, and the first sign is a quality metric moving on a day your team shipped nothing.

This is why the evaluation set should run on a schedule and not only in CI. A nightly run against production configuration turns "users started complaining last week" into "quality dropped on Tuesday", which is the difference between an investigation and a guess.

Drift also comes from your own side in ways no deployment records. A reindexed corpus, a changed embedding model, or a document set that grew in one direction will all move behaviour without a release, and only a scheduled measurement catches them.