Running GPUs on Kubernetes means building your own cost visibility

Standard Kubernetes tooling provides per-pod metrics, rightsizing recommendations and cost allocation for CPU and memory out of the box. Shared GPUs strip away these conveniences entirely, leaving platform teams to engineer their own visibility for the most expensive resource on the bill.

Most Kubernetes operators maintain a Grafana dashboard showing container resource requests versus actual consumption, making it straightforward to identify overprovisioned deployments. Breaking down usage by namespace reveals which teams consume what, a practice that has worked reliably for CPU and memory for years. GPU workloads shatter this assumption: nearly every visibility mechanism teams rely on requires custom rebuilding.

The challenge cuts deeper because GPUs represent the largest expense. Teams must construct their own metrics pipelines, define accuracy standards, establish guardrails and devise billing formulas. Each layer demands ongoing maintenance and ownership.

Bring your own metrics

The kernel tracks CPU and memory usage per container automatically, making per-pod visibility nearly free. GPUs work differently. NVIDIA's DCGM exporter, the standard starting point, reports metrics at the physical device level by default. This functions adequately when a single pod owns an entire GPU, but GPU sharing through time-slicing or MPS breaks the model. When three pods share one GPU, DCGM reports identical values for all three, summing to 300% of a GPU that does not exist. The driver exposes only a single counter set for the whole GPU, leaving DCGM unable to attribute usage to individual workloads.

The resulting numbers plot smoothly on dashboards and support polished reports, yet teams build visibility on fundamentally incorrect values that remain difficult to detect.

Bring your own rightsizing math

CPU resource management relies on millicores, overcommitting, burst capacity and throttling limits that never trigger OOM kills. GPUs operate as whole-number resources: nvidia.com/gpu: 1. Requests and limits must match exactly, and the scheduler enforces no overcommitting. Fractional sharing for inference workloads happens beneath the scheduler through time-slicing or MIG, mechanisms that do not integrate cleanly with Kubernetes resource requests.

The fundamental rightsizing exercise—comparing requested capacity against actual consumption to recover waste—barely exists for GPUs initially. Teams must first determine what fraction of GPU capacity a workload received before any waste conversation becomes possible.

Bring your own guardrails

The NVIDIA container runtime checks an environment variable called NVIDIA_VISIBLE_DEVICES, and base CUDA container images ship with it set to all. Normally the device plugin overrides this, injecting only the GPUs Kubernetes allocated to a pod. If nothing overrides the setting or the runtime honors the container's environment variable, a pod can access and claim every GPU on the node regardless of Kubernetes allocation. What the scheduler recorded as allocated can diverge significantly from what actually runs on hardware. Determining the real allocation requires examining process-level activity and reconciling it against Kubernetes records.

Bring your own definition of utilization

A CPU graph showing 80% utilization conveys clear meaning. GPU utilization proves far more ambiguous. The standard metric indicates whether a kernel ran on the GPU during a sample window, but reveals nothing about actual compute or memory bandwidth consumed. GPUs frequently sit at 100% utilization while performing minimal computation. Memory often becomes the bottleneck first, since frameworks like vLLM reserve most GPU memory at startup regardless of actual need. Teams must identify early which signal matters for their workloads—compute occupancy, memory footprint, bandwidth or power consumption. Relying on the headline utilization number risks reaching incorrect conclusions.

Bring your own attribution

NVIDIA's NVML library reports per-process GPU memory consumption and enables tracing each process to its owning pod and team. The numbers prove reliable with one caveat: they do not sum to the card's total because the driver reserves memory for itself outside any workload.

Compute attribution falls apart, and understanding why matters. Consider a shared kitchen: memory resembles counter space where you can see what ingredients sit where. Compute resembles a stove where everyone takes millisecond-duration turns. NVIDIA provides not "how much stove time did each cook get" but rather a flag checked many times per second asking "is this cook doing something right now?" When two people work simultaneously, both flags read yes almost constantly, producing two jobs that each appear 90% busy—true yet uninformative about who accomplished more.

This measurement works in one scenario: when the GPU strictly alternates between jobs one at a time, where tools like DCGM's accounting mode or nvidia-smi PID-level sampling can approximate each job's share. It fails when jobs genuinely run side by side—precisely the setup teams choose for better utilization. If two teams share a GPU, no available tool today reveals which one consumed 70% and which consumed 30%.

The practical solution involves billing on memory, where data remains accurate, and treating compute as a signal for chargeback purposes. Fairness through accuracy between teams matters when moving beyond showback, and GPU memory represents the most accurate attribution method with existing tools.

Bring your own splitting rules (and the bill is brutal)

GPU sharing encompasses several distinct approaches, each altering the underlying mathematics. Time-slicing round-robins the GPU without isolation and permits oversubscription. MPS allows kernels from different processes to run simultaneously. MIG splits the GPU into fixed hardware partitions and enjoys popularity, though most cloud providers charge for the entire physical GPU even when using only a slice, and A100/H100-class GPUs support only up to 7 instances. DRA, Dynamic Resource Allocation, has matured across recent Kubernetes releases and represents the clear direction forward, but the surrounding ecosystem including autoscalers like Karpenter continues catching up. Each approach redefines what "allocated" means, so visibility depends on which mode a node runs.

GPU costs dwarf CPU costs per hour and resist per-workload splitting. Cloud providers bill GPU nodes as single line items combining CPU, memory and GPU. Reaching a real dollar figure per workload requires separating that line and weighting the GPU piece by model, since a T4 and an H200 command vastly different prices. Then the GPU cost splits across workloads sharing it. On AWS, the cheapest MIG-compatible A100 GPU costs $22/hr, roughly $14/hr on spot, while a comparable CPU-only instance of equivalent size runs about $5/hr. Even on spot, the difference proves substantial.

So what does this actually cost you?

The true expense extends beyond the GPU bill itself to encompass engineering time and operational burden spent reconstructing visibility previously obtained at no cost. Metrics pipelines, accuracy standards, guardrails, attribution logic and splitting formulas each represent a layer teams must now own and maintain.

The encouraging news: the space evolves rapidly. DRA matures and adoption grows with it, hardware partitioning becomes easier, and tooling gaps narrow with each release. Teams should make deliberate choices about which layers to build now versus which to defer while the ecosystem catches up. Critically, do not assume that CPU and memory strategies transfer directly to GPU workloads.