Huyen's Inference Optimization Playbook Holds Up a Year Later

Machine learning researcher and author Chip Huyen is heading back to P99 CONF on October 21–22, bringing fresh perspective to her guidance on managing AI inference expenses in an era dominated by agentic systems.

A year ago, Huyen delivered a keynote at the same conference—an event that brings together developers working on high-performance, low-latency systems. The author of the bestselling AI Engineering opened her remarks with a straightforward observation: while training a frontier model represents a one-time expense, inference costs accumulate repeatedly throughout a model's lifetime. Across the lifespan of a typical model, Huyen estimates the computational balance between training and inference falls somewhere between 1:10 and 1:100. Reasoning models, which consume far more tokens, push that ratio even wider. This economic reality matters because if inference becomes prohibitively expensive, the initial investment in training never gets recouped—a dynamic that has spawned countless jokes about frontier model profitability.

Huyen spent months researching optimization strategies for her book, then condensed her findings into a 30-minute conference presentation in October 2025. Now, with a year of industry evolution behind that talk, it's worth revisiting what she covered and how those ideas have held up.

Measuring what matters

https://www.youtube.com/embed/-MIv3mWAlBc?feature=oembed

Huyen advocates tracking several core performance indicators:

• Time to first token (TTFT): The delay before a user sees any output
• Time per output token (TPOT): The interval between consecutive tokens, also called inter-token latency
• End-to-end latency: Time to first token plus time per output token multiplied by the number of output tokens minus one

With reasoning models, a complication arises: some generated tokens never reach the user. "The first generated token might not be the same as the first visible token," Huyen explained. "The model might think for a while, and it will only show the first token of the final output to the user." Some practitioners measure Time to Publish instead—the moment when the first visible token appears on screen. The right metric depends on what your users actually care about.

Beyond throughput—the raw count of requests processed in a time window—Huyen recommends tracking goodput: requests that actually met your performance targets. Her example: an application aims for 200 milliseconds time to first token and 100 milliseconds time per output token, processes 10 requests per minute, but only three satisfy both criteria.

Three angles for optimization

Inference servers offer levers across three dimensions: hardware, the model itself, and the service orchestrating requests and responses.

Huyen, who previously held a position at Nvidia, deliberately sidestepped hardware considerations. "Even though I find it to be an intellectually interesting topic, it's not relevant to a lot of people because we don't have the power to change the hardware itself," she noted. She also downplayed replica parallelism—simply adding more machines—as a solution, since it grows expensive and complex quickly, particularly when managing a heterogeneous fleet of 80GB, 48GB, and 24GB devices alongside models of varying sizes.

That leaves model-level and service-level strategies. Huyen frames the choice this way: "If you want to host the models yourself, or if you have access to the model weights, or if you train a model yourself, or you want to fine-tune or distill a model, then model optimizations might be for you. However, if you want to take a model as-is and make it more efficient on your own inference service, you might want to look into service optimizations."

Model optimization

These methods alter the actual weights and can shift model outputs.

Quantization reduces the bit depth used to represent weights and activations—for instance, dropping from 32-bit (four bytes per parameter) to 8-bit (one byte). Huyen elaborated: "Reducing the precision not only reduces the memory requirement to run the model, making it cheaper. It can also make the model a lot faster. If you do additions bit by bit and each weight is 32 bits, you have to do it 32 times. If it's 8 bits, you only have to do it eight times." The tradeoff involves a modest quality loss. She continued: "It's possible to reduce a lot of the model's memory footprint with minimal quality degradation, and quantization is pretty generalizable to a wide variety of model architectures and model sizes. That's why it's very popular. I rarely see any companies running a model at full precision anymore."

Distillation leverages a large model to produce training examples for a smaller one. Suppose you have a powerful model (Huyen cited o1 as an example) and want a compact version with similar performance. You gather a substantial set of prompts, feed them through the larger model, then train the smaller model on those outputs.

Caution is warranted, though. Huyen warned: "A lot of model providers have the condition that they do not allow their models to be used to train competitive models. So even though it's a very common technique, you need to check licensing."

Service optimization

These techniques govern how requests are scheduled, routed, and reused—the weights themselves remain unchanged.

Batching groups multiple requests for processing in a single model pass, far more efficient than handling them individually. Huyen outlined several approaches:

• Static batching waits for the batch to reach capacity, maximizing compute use but potentially raising latency for early requests
• Dynamic batching operates on a timer (say, every 15 milliseconds), sacrificing some compute efficiency for better latency
• Continuous batching addresses the reality that LLM requests finish at vastly different times—one asks for Vietnam's capital, another launches extensive research. Under static or dynamic batching, a finished request's slot idles until the slowest completes, while new requests pile up. Continuous batching returns each request as it finishes and fills the vacancy with another, boosting both resource use and latency

Decoupling prefill and decode separates these two request phases onto distinct machines. Prefill handles the input; decode generates the output. Huyen noted: "Input tokens can be processed in parallel, whereas output tokens need to be generated sequentially. With parallel processing, it's bounded by compute, the processing power of the chip. With decoding, it's bounded by memory, because you have to move model weights." Since each phase demands different resources, most services now isolate them. To reduce time to first token, allocate more capacity to prefill. To improve time per output token, shift resources to decode.

Parallelism distributes computation across machines. Replica parallelism copies the entire model to additional machines. Tensor parallelism breaks a large matrix so different machines compute different sections. Pipeline parallelism divides the model by layer, allowing requests to flow through as a pipeline.

Prompt caching processes shared text segments once, then reuses them, cutting both cost and latency. Repetition abounds across requests to the same application: system prompts, examples, identical codebases, the same document with different questions. Processing that shared portion once and caching it makes obvious sense.

When Huyen was writing AI Engineering, this technique was relatively obscure. "There was one paper about it, and it was not really known, but it made a lot of sense. So I included prompt caching in the book, and I'm very happy to see that nowadays it's pretty much everywhere." Savings depend on how much of your prompt gets cached. Huyen's open-source tool Sniffly observed cache hit rates of 90% to 97% in Claude Code logs. Some providers rewrite prompts internally to boost hit rates, but you can structure them yourself. Huyen's advice: "It's pretty easy to do, and it can improve your application performance significantly." Since caching works on shared prefixes, position stable prompt elements first and variable ones later.

Choosing an inference provider

Huyen concluded with guidance for those selecting an inference provider: "There are many inference companies that provide inference optimizations for models you want to use, and a lot of them advertise just cost and latency. But pay attention to how many inference optimization techniques also change the model behavior or reduce the model quality. So when evaluating an inference service, it's important to look not just at cost and latency, but also at model quality. Does this model, provided on this service, also perform similarly on standard benchmarks?"

How the advice has aged

A year has passed since that keynote, and most of Huyen's framework has proven durable.

On the economics front, her analysis appears sound. For most practitioners, however, not all the cost levers she outlined are within reach. Still, understanding the underlying mechanisms matters. For someone experimenting with local LLMs, her points on parallelism (often unavailable) and prompt caching and quantization (more accessible) ring true.

Prompt caching, which Huyen described as novel when she wrote her book, is now bundled into every API token package offering. Her observation about Claude Code achieving 90% cache hit rates likely explains why individual developers can still afford agentic coding agents.

Some predictions have evolved in unexpected directions. Huyen flagged reasoning models as making inference even more consequential. A year later, agents executing multi-step loops with tool calls have transformed that observation from a minor note into a major force—one that can flip Claude's rate limits and their notorious 99.x% availability into advantages.

The metrics Huyen outlined—time to first token, time to publish, goodput under a latency SLO—have entered standard industry vocabulary and likely demand fresh analysis in light of recent developments.

Those topics may well feature in this year's presentation. A complimentary pass to P99 CONF 2026 is available, with sessions running October 21 and 22.

Source: The New Stack