Kubernetes doesn't scale with a single number: three different bottlenecks

When someone asks how much the Kubernetes control plane needs to grow, the answer usually comes in the form of a node count. It's a convenient metric, but a misleading one: there are small clusters that suffer much more than others ten times larger. The key lies in what type of pressure they withstand. And there, adding apiservers blindly can be the worst decision.

Three clusters, three problems that don't look alike

Imagine three different scenarios. The first is a training fleet that had been running for months on a few thousand nodes and suddenly jumps to fifteen thousand: the kube-apiserver starts consuming memory and etcd complains. The second never exceeds a few thousand nodes, but its objects are alive: sessions that are created, suspended, resumed and destroyed thousands of times per minute, and every transition is a write. The third is neither large nor fast: a platform cluster with two hundred operators and tens of thousands of objects that mostly do nothing, but every apiserver keeps a copy of all of them.

Two of those three cases are AI workloads and one is not, but that is the least relevant part. What matters is that they are running out of headroom in different directions, and fixing one does nothing for the others. The original article sums it up like this: scaling isn't the hard part, what's missing are the assembly instructions.

The three dimensions of scaling

The first axis is the per-node limit. Training and reinforcement learning fleets: large, homogeneous, and the pressure comes from what each kubelet reports. With one hundred thousand nodes and a heartbeat every ten seconds, the liveness check alone generates about ten thousand writes per second; in the article's measurements, lease renewals accounted for close to 70% of write QPS. Most of what that control plane does is have the nodes politely say "I'm still here," endlessly.

The second axis is the churn limit. Agent platforms: modest size, very intense day. Systems that pack many agent sessions into a smaller group of worker pods express each actor's lifecycle as a custom resource, so the busiest and most ephemeral objects are CRDs, and the write rate has nothing to do with how many machines you have.

The third is the state limit. Platform clusters: many types, many objects, most of them dormant. The cost isn't the writes, but the resident bytes multiplied by every apiserver that stores them. This is the axis where the reflex of adding an apiserver really hurts.

Why adding apiservers can make things worse

When the control plane gets stuck, the usual response is to add apiservers. It's an option documented in kOps and what most people will answer. But every apiserver you add replicates the full watch cache: you buy request capacity and pay for it with memory you didn't have. In a state-limited cluster, it's shooting yourself in the foot.

The article proposes two useful concepts. Seams are the points where you can take Kubernetes apart: a contract between two components, usually at a network boundary, replaceable without patching upstream. Like the seams of a jacket: you can adjust it at the waist or widen the shoulders and it's still the same jacket. If you cut where there is no seam, you're left with an expensive rag. Recipes are configurations proven at scale, executable, that pass or fail their scale test, with CI warning the day they stop being valid.

The four seams worth knowing

1. The storage contract: which store holds which resource types. It relieves churn. Not all state deserves the same durability guarantees: events and node leases are ephemeral, and leader election ones too, and none of them needs the same Raft-replicated store as your Deployments. KEP-4743 defines that boundary and --etcd-servers-overrides already allows routing resource types to different backends. The limitation: it only knows built-in resources, so a churn-limited platform can't route the CRDs that generate its write storm.
2. The node liveness contract: how liveness reaches the control plane. It relieves the node count. Two fixes that compose: absorb it with a node-oriented layer in front of the control plane that manages renewals and propagates only real transitions, and make it cheaper by routing renewals through etcd's in-memory lease update instead of a full Raft commit. That second change lands in etcd and helps any cluster, even a three-node one.
3. The sharding axis: the field by which controllers, caches and stores accept splitting. It relieves all three axes. KEP-5866 introduces hash-range filtering for API traffic, which is the foundation; coordination, resharding and controller manager sharding are out of scope and need work.
4. Cache residency: what an apiserver actually keeps in memory. It relieves state. An apiserver that only talks to the scheduler doesn't need to have all objects resident, and a two-shard prototype reduces the heap by roughly half per shard. There is a floor: with all disableable caches turned off, an apiserver still loads about 0.9 KB per RBAC object, because the authorizer's informers retain them anyway.

What it means for an SMB and what we recommend

Most SMBs won't have fifteen thousand nodes, but they can fall into the third pattern without realizing it: a platform cluster with many operators, many objects and several apiservers replicating cache. And there the reflex of scaling vertically or adding replicas makes the bill more expensive without solving anything.

Our practical recommendation:

• Measure before touching. Distinguish whether your bottleneck is nodes, churn or state. Without that classification, any adjustment is a gamble.
• Review what you store in etcd. Ephemeral events and leases shouldn't compete with business state, if your version and distribution allow it.
• Don't add apiservers out of inertia. In state-limited clusters, every new replica is memory that multiplies.
• Watch the operators. Two hundred operators in a small cluster generate more pressure than it seems; install them judiciously and review the ones that no longer contribute.
• Design for the real peak, not the average. As we told in the engineering lessons for traffic peaks, behavior under extreme load reveals what day-to-day hides.

The conclusion is simple: Kubernetes scales, but not with a single number. Understanding what type of pressure your control plane withstands is what separates a useful expansion from a higher bill with the same problems. If you want to review how your cluster is set up before the peak arrives, at ForgeNEX we can help you diagnose it.

Source: The New Stack. Analysis and adaptation: ForgeNEX.