Measure Before You Buy: A Framework for Right-Sizing GPU Capacity in Kubernetes

When GPU pools reach full allocation and batch queues grow, the instinct to purchase more hardware can feel inevitable. Yet a lengthening queue may signal scheduling inefficiencies, insufficient serving capacity, or mismatched model and hardware choices rather than genuine capacity shortages. Before committing to additional GPUs, engineering teams need to establish what success means, when results must arrive, and what the delivery cost should be.

The foundation of this analysis rests on a critical distinction: "An allocated GPU, a busy GPU, and a workload producing valuable results are not the same thing." Recognizing how these states differ enables teams to extract more value from existing resources before acquiring capacity they may not actually require.

Start with the outcome and deadline

Workload urgency cannot be determined by broad categories alone. A customer service chatbot demands continuous inference availability, while a model evaluation job might only need results before the next internal synchronization meeting. Similarly, some batch inference tasks tolerate delays, yet certain training jobs carry firm completion deadlines.

For every workload, teams should record its deadline, expected runtime, resource needs, and whether it can tolerate interruption. Equally important is defining what constitutes valuable output. An inference service requires both a task-specific quality threshold and a latency target. A finished request or properly formatted JSON response alone does not confirm the workload achieved its objective.

Distinguish between flexibility in start time and tolerance for interruption. A job capable of waiting to begin may still require uninterrupted execution once started.

Use scheduling flexibility before expanding the pool

Consider a scenario where four GPUs become available for batch work between noon and 5 p.m. Four independent evaluation jobs are queued at noon, each requiring two GPUs for one hour, with all results due by 5 p.m.

Launching all four jobs simultaneously would demand eight GPUs. However, running two jobs at noon and two at 1 p.m. completes all four by 2 p.m. using only the four available GPUs. Assuming runtimes and supporting infrastructure remain constant, both approaches consume eight GPU-hours of total capacity, excluding initialization overhead.

"Staggering reduces peak demand. The jobs still finish on time without expanding the pool." Before provisioning additional GPUs to clear a backlog, evaluate how much flexibility exists in job start times.

Tools such as Kueue establish an admission control layer, applying configured quotas and policies to determine when work enters the system. Teams must still decide how to represent and enforce deadlines and resource specifications. Flexible queueing mechanisms can support capacity planning and prevent the costly pattern of starting jobs early only to interrupt them later.

Measure how much useful work the deployment can sustain

Flexible scheduling addresses capacity constraints, yet inference teams must also assess whether the model and serving setup match the actual workload.

"High GPU activity does not establish how much additional throughput is available." According to Nvidia's profiling documentation, streaming multiprocessor activity can mask warps waiting on memory operations. Serving metrics such as active requests, queued requests, and cache occupancy provide essential context, while preemption counts illuminate system behavior. In vLLM, KV-cache usage indicates the fraction of occupied cache blocks but does not directly reveal total serving capacity.

Establish sustainable capacity through a representative load test. Hold the model, hardware, serving parameters, and request composition constant. Gradually increase the arrival rate and track throughput, quality, and latency. The maximum sustained rate that still satisfies the workload's requirements—including error thresholds and chosen latency percentiles—yields a meaningful capacity baseline. For interactive generation, measure time to first token and subsequent token delivery, not merely average completion time. DistServe demonstrates why serving capacity evaluation must align with application-level objectives such as latency.

Compare actual production traffic against this tested capacity. Low cache occupancy may warrant deeper investigation, but claiming ten times greater throughput demands a throughput test. Temporary queues can reflect brief traffic spikes, while sustained queues may point to underlying problems. Neither pattern alone identifies the bottleneck.

Connect OpenCost data to verified outcomes

The FinOps community approaches AI efficiency through the lens of cost and tangible business results. For AI workloads, cost becomes more meaningful when connected to the outcomes the infrastructure produces. Kubernetes teams can make this relationship explicit by merging workload cost, serving behavior, and application results.

Leverage OpenCost's allocation API to extract workload costs at specified intervals, then correlate that data with serving telemetry and outcome records using workload identities and timestamps. OpenCost also offers opt-in inference cost tracking, including cost per million tokens. When computing cost per outcome, include applicable GPU, CPU, memory, and other infrastructure expenses, and clarify how shared and idle costs are attributed. Incorporate relevant costs outside OpenCost separately to avoid double-counting.

One valuable metric is cost per verified outcome: total attributable cost divided by results meeting the defined quality criteria and delivered within the required window. Factor in retries, failures, startup time, and idle periods where they contribute to total cost. Count the final business result once. When correctness relies on sampled reviews, document the sampling approach and associated uncertainty.

OpenCost provides cost allocation details; the application supplies outcome definition and evaluation evidence. Capacity measurement remains a separate concern. OpenCost proposal 3915 explores integrating serving-engine telemetry into the cost model, though it remains in draft form rather than released as a saturation or correctness score.

Test whether the spare capacity earns its cost

Imagine a service handling 1,000 tasks hourly with a 98% correctness requirement. A small model achieving 90% accuracy is unsuitable, regardless of cheap GPU costs or high utilization.

A more capable model might deliver 990 correct and timely results for $2 per hour while consuming only a fraction of its tested serving capacity. Its cost works out to roughly $2.02 per 1,000 qualifying outcomes, and that unused capacity may represent an acceptable price for meeting quality standards and handling traffic bursts.

The next step is testing whether a smaller GPU, fewer replicas, or different batching configuration can maintain those outcomes and required reliability. If a tested setup produces the same 990 qualifying results for $1.50, the cost drops to about $1.52 per 1,000 qualifying outcomes. Savings emerge from delivering identical outcomes with fewer resources. Pushing utilization higher on the original GPU without improving outcomes provides no comparable benefit.

When multiple models meet minimum requirements, additional quality gains may justify higher costs if they reduce consequential errors or human review needs. Weigh that value against actual demand. Cost per correct outcome proves most useful when compared outcomes carry similar value and error consequences.

Adjust one significant variable at a time, then compare cost, verified outcomes, latency, and deadline achievement under representative load. Preserve capacity for traffic spikes and failures, and confirm that low activity reflects genuinely available capacity. A GPU assigned exclusively to one Pod cannot serve another until released.

"The target is a deployment that delivers the required outcomes at a defensible cost, with enough headroom to meet its requirements." Teams can then select the change evidence supports: defer flexible jobs, enhance batching, adjust deployment size, switch models, or purchase additional GPUs.