What an AI inference engineer does

Training a model is a large one-off expense. Serving it is a recurring one that grows with every user. For any company with real traffic, inference cost eventually dominates, and at that point a percentage improvement in efficiency is worth a great deal.

That creates demand for people who can make a model produce the same output using fewer resources, or the same resources serving more requests. The appeal of the work is how legible it is: you reduced cost per request by a third, or you did not.

It is also unusually defensible work in a field where much of the tooling changes every year. The constraints — memory bandwidth, batch scheduling, precision trade-offs — are properties of the hardware rather than of whichever framework is current.

It is systems engineering with a machine learning object at the centre. Much of it looks like conventional performance work — profiling, finding the bottleneck, removing it, measuring again — applied to accelerator hardware and model execution rather than to a web service.

The recurring theme is trade-offs against quality. Nearly every technique that makes inference cheaper risks changing output, and the job is knowing how much change is acceptable for how much saving. This is why inference engineers and evaluation engineers work closely together.

A surprising amount of the highest-value work is not exotic. Routing easy requests to a smaller model, or caching a shared prefix that every request repeats, frequently produces larger savings than any amount of kernel-level tuning, and neither requires deep hardware expertise.

• Batching strategy — grouping requests without breaking latency targets
• Quantisation — lower numerical precision for speed and memory, at some quality cost
• Caching — reusing computation across requests that share a prefix
• Model selection and routing — sending easy requests to a smaller model
• Serving infrastructure — autoscaling expensive hardware without waste
• Hardware choice — matching accelerators to actual workload shape

Inference has its own vocabulary, and using it correctly in an interview signals immediately whether you have done this work. Time to first token and inter-token latency matter separately because they affect perceived responsiveness differently in a streaming interface.

The central tension is between throughput and latency. Larger batches use hardware more efficiently and raise throughput, while making individual requests wait longer. Nearly every meaningful decision in the role is a position taken on that curve.

Utilisation is the metric that reveals the most about a deployment. Expensive accelerators sitting idle between bursts is the most common and most expensive failure, and it is an infrastructure problem rather than a model one.

• Time to first token — how quickly a response begins
• Inter-token latency — how smoothly it continues
• Throughput — total tokens served per unit of hardware
• Cost per thousand tokens, or per request
• Accelerator utilisation — whether expensive hardware is actually busy

You do not need to be able to train a frontier model. You do need to understand enough about how models execute to know what is expensive and why — memory bandwidth as the usual constraint rather than raw compute, why sequence length affects cost non-linearly, what a key-value cache stores and why it dominates memory.

Around that sits ordinary but high-quality systems engineering: profiling, memory management, concurrency, and the ability to reason about a distributed serving tier under load.

The single most useful mental model is that generation is usually bandwidth-bound rather than compute-bound. Once that is internalised, most of the techniques stop being a list to memorise and become obvious consequences of moving less data.

• Transformer execution at a mechanical level, not a mathematical one
• GPU fundamentals — memory hierarchy, bandwidth limits, occupancy
• A production serving framework, learned properly rather than skimmed
• Profiling and benchmarking with results you can defend
• Quantisation methods and their measured quality cost

Backend and systems engineers with performance experience transfer most naturally, because the discipline of profiling before optimising is the whole method and it is already habitual for them. The model knowledge required is narrower than it looks from outside.

People from an ML research background sometimes struggle initially, not for lack of understanding but because research rewards accuracy at any cost while this role rewards acceptable accuracy at minimum cost. That is a different instinct and takes adjusting to.

Game and graphics engineers are an underrated source of talent here. GPU memory hierarchies, occupancy and frame-budget thinking map almost directly onto inference work, and the habit of hitting a hard per-frame deadline is exactly the right instinct for a latency target.

The most expensive pattern is optimising the model while the infrastructure idles. Teams spend weeks on quantisation and kernel selection while their accelerators sit unused between traffic bursts, which costs more than every technical gain combined.

Measure utilisation before touching the model. If expensive hardware is busy a third of the time, the first problem is scheduling and autoscaling, and solving it requires no model knowledge at all.

The second most expensive pattern is optimising without an evaluation suite. A change that cuts cost by forty per cent and degrades quality in a way nobody measured is not a saving, and it is usually discovered weeks later by a user rather than by the team.

This is one of the easiest AI specialisms to demonstrate without a job in it, because the results are numbers. Take an open-weight model, serve it, measure a baseline, apply techniques and report what changed — including the quality cost, not only the speed gain.

A write-up showing a benchmark, a change, a result and an honest note on what degraded is a stronger portfolio piece than most professional experience described vaguely. Interviewers in this field respond to measurements.

Include the change that did not work. Reporting that a technique you expected to help produced no improvement, and explaining why, demonstrates that you measured rather than assumed — which is the habit the role is actually hiring for.