Before expanding your GPU pool, measure the actual outcome

When a team runs AI workloads on Kubernetes, seeing the GPU pool fully occupied and a batch job queue that keeps growing often leads to a quick conclusion: capacity must be expanded. But that queue may be signaling scheduling problems, service capacity issues, or poor model and hardware choices, not necessarily a lack of cards. Before provisioning more capacity, it is worth being clear about what success means, when it must be finished, and how much it costs to deliver.

The core idea is simple but often forgotten: an allocated GPU, a busy GPU, and a workload that produces valuable results are not the same thing. Understanding how these three concepts relate allows you to make better use of existing capacity before buying the capacity you actually need.

Start with the outcome and the deadline

The first step is to look at the workloads and the use cases they support. A customer service chatbot may need inference capacity around the clock, while the results of a model evaluation job may only be needed before the next internal release sync. Classifying by broad categories such as "inference" or "training" does not determine urgency: there is batch inference that can wait and training with a firm deadline.

For each workload, it is worth documenting the deadline, expected runtime, resource requirements, and whether interruption is tolerated. You also need to define what counts as a valuable output. An inference service should have a quality threshold specific to its task and a latency target; a request finishing or returning valid JSON does not prove it has met its goal.

An important nuance: flexibility to start is not the same as tolerance for interruption. A job that can wait to start may need, once launched, to run without interruption.

Use scheduling flexibility before expanding

Let's take a concrete example. Four compatible GPUs are available for batch work from noon to five in the afternoon. At noon, four independent evaluation jobs are ready. Each needs two GPUs for one hour, and all results are due by five. Launching all four at once requires eight GPUs. If they are staggered two at noon and two at one, everything finishes by two using only the four available GPUs.

If runtimes and auxiliary resources do not change, both plans consume the same GPU hours. The difference is that staggering reduces peak demand and the jobs still finish on time without expanding the pool. Tools such as Kueue provide an admission control point that, with configured quotas and policies, decides when a job is admitted. The team still has to decide how to represent and enforce deadlines and requirements.

Measure how much useful work your deployment sustains

Flexible scheduling relieves pressure, but in inference you also need to review how the model and serving configuration fit the actual load. High GPU activity does not guarantee that performance headroom remains. Nvidia's profiling documentation notes that streaming multiprocessor activity may include warps waiting on memory. Service metrics such as running requests, waiting requests, and cache occupancy provide context, and preemption counters help explain behavior. In vLLM, KV cache usage measures the fraction of occupied blocks, but it does not directly measure total serving capacity.

To estimate sustainable capacity, you need a representative load test: keep the model, hardware, serving settings, and request mix fixed, increase the arrival rate, and observe throughput, quality, and latency. The highest sustained rate that still meets the requirements—including error limits and latency percentiles—gives a useful estimate. In interactive generation, include time to first token and subsequent delivery, not just average completion time.

Short queues may be temporary spikes; persistent ones may require diagnosis. Neither one by itself identifies the limiting resource.

Connect cost with verified outcomes

The FinOps community frames AI efficiency in terms of cost and useful outcomes. In Kubernetes, this can be made concrete by combining workload cost, service behavior, and application results. OpenCost's allocation API allows you to obtain per-workload costs at defined intervals and cross-reference them with service telemetry and outcome records using identities and timestamps. OpenCost also offers optional inference cost tracking, including cost per million tokens.

When calculating cost per outcome, include GPU, CPU, memory, and other infrastructure costs, and document how shared and idle costs are allocated. A useful metric is cost per verified outcome: total attributable cost divided by the outcomes that meet the required quality and arrive on time. Count retries, failures, startups, and idle periods when they contribute to the total, and count the final business outcome only once.

What we recommend at ForgeNEX

For an SME or an IT team with limited resources, the temptation to expand hardware is understandable but expensive. Our recommendation is to treat the GPU pool like any other critical resource: with business metrics, not just utilization metrics.

• Define the outcome before buying. For each workload, put in writing the deadline, expected time, resources, and tolerance for interruptions. Without this, any expansion is blind.
• Stagger before scaling. Check whether queues can be resolved by shifting job start times. Admission tools such as Kueue can help if the volume justifies it.
• Run a representative load test before claiming you need more capacity. GPU activity is misleading.
• Link cost and outcome. OpenCost provides the cost side; you have to provide the definition of quality and deadline. Without both, the data is not useful for decision-making.

In a context where IT talent is scarce and every infrastructure euro competes with other priorities—as we discussed when talking about IT salaries and the lack of profiles—squeezing existing capacity before expanding it is not a minor optimization: it is responsible management. And if your team is also dealing with supply chain security, it is worth keeping an eye on cases like the VeloCloud zero-day, where a read-only account is enough to compromise the environment.

Expanding a GPU pool without measuring the outcome is an expensive way to postpone a conversation your team should be having anyway: what work adds value, by when, and at what cost.

Source: The New Stack. Analysis and adaptation: ForgeNEX.