Healthy Kubernetes but requests stall: what to look at

There is a situation that drives any operations team to despair: the Kubernetes dashboard is green, the pods are running, the liveness and readiness probes return success, and the CPU of the nodes is nowhere near its limit. And yet, users open tickets because the application is intermittently slow. How is it possible that basic checks pass while performance degrades? This is what the industry calls phantom slowdown, and understanding it is key for any small business that already has workloads in containers.

Why a "healthy" cluster does not mean the app works

A liveness probe is used to know whether to restart a container, even one whose process is alive but blocked. A readiness probe indicates whether the container can receive traffic. Custom checks can include conditions of the application or its dependencies, but passing a basic probe does not equal a complete request meeting its response time goal. They are two different things and should not be confused.

Mahesh Ramachandran, head of product for HPE OpsRamp Software, explains it to The New Stack with a composite scenario: users of an application that serves AI prompts on Kubernetes report sporadic slowness. The team reviews cluster, nodes, and pods, everything seems correct, and there are no pod failures. The problem is precisely that intermittency: without historical evidence, locating the origin is almost impossible.

"When degradation occurs intermittently, pointing to the cause without historical evidence becomes almost impossible."

The practical conclusion is simple: passing the probes narrows the investigation, it does not close it. A snapshot of a healthy cluster may not capture a previous episode of slowness, and the absence of container failures does not prove that the application or its dependencies are doing well.

Continuous telemetry across the stack: separating signal from noise

To solve sporadic slowness, you have to go beyond superficial Kubernetes metrics and deploy continuous telemetry at various layers. A user request may depend on the application code, the container runtime, virtualized or bare metal infrastructure, and public cloud APIs. The relevant layers depend on each one's deployment.

According to Ramachandran, continuous collection covers nodes, pods, ingress controllers, and services, capturing logs, events, and metrics. The foundational step is to have a continuous record: you have to go back to the history to correlate the symptoms reported by the user with the actual evidence. But finding that evidence is usually not trivial.

"An anomaly may not appear as an obvious spike in a single metric. It can be a subtle variation or a correlated behavior where two signals seem normal separately, but their combination produces latency."

This is important for a small business: often the problem is not in a graph that spikes, but in the correlation of two signals that, seen separately, seem correct.

How to investigate a latency episode, step by step

1. Start from the affected transaction and the specific time window.
2. Compare response times and queries per second with the health and number of pods.
3. Look for variations in the collected signals that coincide with the slowdown.
4. Treat the correlation as a hypothesis, not as a proven cause.
5. After applying a change, compare the same user-facing measures with the previous behavior.

Coverage depends on the agents, integrations, permissions, and instrumentation that each team configures. In the case of OpsRamp, OpenTelemetry-compatible traces are collected for instrumented applications (note: without instrumenting the app, infrastructure monitoring does not generate traces), cluster and workload telemetry through Kubernetes agents, virtualization and HCI data through integrations with VMware, Nutanix, and HPE Morpheus VM Essentials, and physical server telemetry via Redfish with the corresponding accesses and gateways.

What this means for your small business

If you have applications on Kubernetes—or are migrating—this article leaves several lessons applicable without needing to buy an enterprise suite:

• Do not trust only the green. Probes are a traffic light for availability, not performance. You need user experience or end-to-end latency metrics.
• Keep history. Without retention of metrics, logs, and events, an intermittent problem is undetectable afterward. Adjust retention according to criticality.
• Instrument the application. If there are no traces, there is no way to follow a request through its dependencies. OpenTelemetry is a reasonable standard to start with.
• Correlate layers. The bottleneck may be outside the cluster: in the database, in an external API, or in the hypervisor.
• Validate changes. A correlation is not a cause. Measure before and after with the same yardstick.

At ForgeNEX we see many small businesses that have made the leap to containers without setting up the corresponding observability layer. It is a pattern we already addressed when talking about GPU visibility and costs in Kubernetes, and it repeats here: without continuous and well-correlated telemetry, any intermittent incident becomes a witch hunt. Our recommendation is to start with the basics—metric retention, traces of critical paths, and alerts based on user latency—before considering more sophisticated tools. AI can help sift through the noise, but only if you have first decided which signals you keep and how long you retain them.

Source: The New Stack. Analysis and adaptation: ForgeNEX.