GPU at 100% doesn't mean your AI is doing well: what to measure

There is a scene that repeats itself in many teams that have just deployed an AI service: the Kubernetes dashboard is green, the GPU shows 100% utilization, and everyone breathes a sigh of relief. However, users tell a different story: responses that take too long, answers that arrive quickly but are incorrect, or erratic behavior under load. The problem is not the infrastructure itself, but the lack of a clear connection between what the business sees and what happens in the layers of Kubernetes, GPU, compute, storage, and network.

At The New Stack they have addressed this issue with voices from HPE, and the conclusion is useful for any small business or IT team that is setting up AI inference: a busy GPU is a signal, not a diagnosis. The real operational challenge consists of measuring the inference service before a customer depends on it and then comparing production behavior with that baseline.

Measure the experience, not just the hardware

For a conversational service, the first thing to decide is what experience you want to offer. From there, the metrics are chosen. Among those cited in the news article are time to first token, time per output token, reliability, capacity, resource efficiency, cost, and response quality. Not all weigh the same in every case, but it is advisable to have them documented.

An important detail: a fast first token can mask slow subsequent generation. And a fast complete response can be inaccurate or unhelpful. These are different questions and require different measurements. Time to first token and time to complete response do not measure the same thing; inter-token latency and time per output token help explain generation speed once the first token has arrived. The team must document where each interval is measured and compare results under sustained load and demand peaks.

Profile before launching

One of the clearest recommendations in the article is to profile the AI service system before customers depend on it. That means testing with representative prompt and context lengths, output volumes, and realistic traffic levels, and recording token throughput, latency, and service-level measures under those conditions. That baseline becomes the reference for detecting changes in production.

Correlating those load measurements with infrastructure performance and capacity makes it possible to distinguish whether the increase in inference demand is real or whether there is resource contention. Without that correlation, it is easy to confuse a usage spike with a bottleneck and make wrong capacity decisions.

Quality deserves its own evaluation

Platform telemetry can prove that a request was served, but not that the response is good. Quality needs representative tests and acceptance criteria defined by data science profiles and domain specialists. This is a point that many small businesses overlook: latency and availability are measured, but not whether the model hallucinates or responds uselessly. In addition, that evaluation should be reviewed every time the model, the prompt, data retrieval, or the serving method changes.

When the GPU is busy but the service is slow

GPU utilization is a useful signal, but it does not identify the bottleneck. From HPE they describe the trap of celebrating that GPUs are at 100% without knowing how the service actually performs. Some checks they propose:

• Whether the inference pods scale with metrics appropriate to the load.
• Whether requests reach the intended model.
• Whether related components can be scheduled when needed.
• Whether the network configuration and the response time of the database or vector database keep up.
• Whether the GPU partition is too small for the model being served.

These are lines of investigation, not universal diagnoses. They must be correlated with the affected requests and the time window before choosing a solution. It is also worth remembering that the Kubernetes Horizontal Pod Autoscaler can use configured custom or external metrics, so it should not be replaced just because CPU is not a good trigger for inference. What should be done is to feed and test the metrics pipeline, the scaling limits, and the available accelerator capacity. No scaling policy makes GPUs appear instantly.

What it means for a small business

For a small business deploying AI on Kubernetes, the practical message is that service observability cannot be limited to looking at the infrastructure dashboard. Before putting an assistant or a model in the hands of customers, it is advisable to define what experience is expected, measure it with realistic loads, and save that reference. Then, in production, compare. And add a separate quality evaluation, with criteria agreed with the business.

At ForgeNEX we recommend treating inference as a business service, not as a laboratory experiment. That means agreeing on experience metrics, automating correlation between layers, and reviewing accelerator capacity in advance. If you are building this type of platform, it may help you to review how we approach security and governance in agent environments in Hydro de AWS: el compilador como red de seguridad para agentes IA and in GitHub se blinda para la era de los agentes: ¿qué implica para tu pyme?.

A GPU at 100% is not a certificate of good service. It is only a clue that must be put in context with the user experience, response quality, and cost.

Source: The New Stack. Analysis and adaptation: ForgeNEX.