GPUs in Kubernetes: Visibility and Costs Under Control

If you manage a Kubernetes platform, you probably already have a Grafana dashboard to monitor resource consumption. With CPU and memory, the kernel does it for you: you request four cores, use half, and see it clearly. But when GPUs come into play, almost none of that visibility is free. Everything you took for granted with CPU and memory must be built for GPU: your own metrics, your own precision, and your rules for splitting the bill. And it hurts more because the GPU is the most expensive line in the budget.

The visibility problem: every pod seems to use the entire GPU

With CPU and memory, the kernel already tracks usage per container, so getting a number per pod is free. With GPU, it's not. The usual tool, NVIDIA's DCGM exporter, reports at the physical device level. This works fine when a pod has a GPU to itself, but as soon as you share a GPU between pods with time-slicing or MPS, DCGM gives the same value to all pods sharing it. If three pods share a GPU, all three appear to use 100%, and adding them up gives you 300% of a GPU that doesn't exist. The driver exposes a single set of counters for the entire GPU, so DCGM cannot distinguish which workload consumed what. The numbers look good on a graph, but teams build dashboards and reports on completely wrong and hard-to-detect values.

Resource tuning: no millicores or overcommitment

With CPU, we're used to millicores, overcommitment, bursts, limits that only throttle or cause OOM kills. With GPU, almost none of that applies. A GPU is requested as an integer resource: nvidia.com/gpu: 1. You request full GPUs, the request and limit must match, and there is no overcommitment in the scheduler. The fractional sharing you need for inference happens below the scheduler via time-slicing or MIG, and Kubernetes requests don't fit well. So the basic tuning premise (comparing requested vs. used and reclaiming the difference) barely exists for GPUs at first. You have to calculate what portion of the GPU capacity a workload received before you can talk about waste.

Security barriers: the danger of NVIDIA_VISIBLE_DEVICES

The NVIDIA container runtime looks at an environment variable called NVIDIA_VISIBLE_DEVICES, and CUDA base images have it set to all GPUs. In a normal environment, the device plugin overrides it and injects only the GPUs assigned to the pod. But if nothing overrides it or the runtime is configured to respect the container's variable, a pod can see and hog all GPUs on the machine, regardless of what Kubernetes thinks it assigned. What the scheduler recorded as assigned and what actually ran on the hardware can diverge significantly. To get the real number, you have to look at the process level and compare it with what Kubernetes thinks it did.

Definition of utilization: don't trust the big number

When a CPU graph says 80%, you know what you're seeing. GPU utilization is fuzzier. The usual metric indicates that a kernel ran on the GPU for a fraction of the sampling window, but not how much compute or memory bandwidth was burned. You can see GPUs at 100% utilization while compute is barely doing anything. And often what runs out first is memory, because frameworks like vLLM hog most of the GPU at startup, whether needed or not. It's worth determining early which signal matters for your workloads: compute occupancy, memory footprint, bandwidth, or power. If you trust the headline, you can draw wrong conclusions.

Attribution: bill by memory, not by compute

NVIDIA's NVML library tells you, per process on a GPU, how much memory it holds, and you can trace each process back to the pod and owning team. The numbers are reliable, with an asterisk: they won't add up to the card's total, because the driver reserves a portion of memory for itself that doesn't belong to any workload. Compute is where it falls apart, and it's worth understanding why. Imagine a shared kitchen: memory is the workspace, you see where the ingredients are. Compute is the stove and everyone takes turns in millisecond bursts. What NVIDIA gives is not "how much stove time each cook had," but a flag checked many times per second asking "was this cook doing something right now?" If two people are working, both flags say yes almost all the time, so you have two jobs that appear 90% busy, which is true but says nothing about who did more. That measurement works in a scenario where the GPU strictly alternates between jobs one by one, where DCGM's accounting mode or nvidia-smi's per-PID sampling can approximate each job's share. It breaks when jobs run genuinely in parallel, which is exactly what you choose to improve utilization. If two teams share a GPU, nothing available today will tell you which burned 70% and which 30%. The practical decision is to bill by memory, where the data holds up, and treat compute as a signal to drive chargeback. Fairness through precision between teams matters when you want to allocate costs fairly.

What it means for your SMB and what we recommend

If you're starting with local AI or GPU workloads on Kubernetes, don't underestimate the complexity of visibility and cost control. At ForgeNEX we recommend:

• Start simple: if your team is small, consider managed alternatives like Amazon ECS, which already self-heals GPUs and instances, before setting up a Kubernetes cluster with shared GPUs.
• Define your attribution strategy from day one: bill by memory, which is measurable, and use compute only as an indicator.
• Set up security barriers: check that NVIDIA_VISIBLE_DEVICES doesn't expose unassigned GPUs. A pod should not see more GPUs than Kubernetes gave it.
• Choose your metrics wisely: don't settle for the big utilization number; look at compute occupancy, memory, bandwidth, or power depending on your workload.
• Plan cost allocation: GPU is the most expensive line; without reliable data, internal chargeback will be unfair and generate conflicts.

If you need help setting up an AI platform on Kubernetes with visibility and cost control, at ForgeNEX we can help. You may also be interested in our article on Local AI for businesses: when it pays off and how to start.

Source: The New Stack. Analysis and adaptation: ForgeNEX.