Kubernetes: hidden failures that bring down the control plane

At ForgeNEX we work with SMBs that rely on Kubernetes to keep their critical services running. Control plane availability doesn't depend only on components being alive, but on how they interact under pressure. A recent article from The New Stack describes three scenarios in which a seemingly correct mechanism becomes the amplifier of a local failure. This isn't theory: these are real cases that can affect any team managing clusters, even moderate-sized ones.

The amplifier effect in the control plane

The central idea is that, at scale, a component doesn't need to go down to contribute to an outage. A valid mechanism can amplify a local defect through shared metadata, synchronized startup jobs, or overly aggressive health checks. For example, an informer gives a controller a local view of cluster state so its workers can proceed without repeatedly querying the API server. This is efficient when controllers start independently. But if many controllers restart at once in a large fleet, rebuilding those views can generate significant pressure on the control plane. The informer mechanism itself becomes the amplifier.

Three cases you should know about

The article examines three specific situations. In the first, an extension API server published an invalid schema after updating a dependency, and blocked clients that had never used its API. In the second, DaemonSet agents restarted across a large fleet, turning routine startup work into a storm of requests that the API server couldn't absorb. In the third, a deep liveness check on a component in the API request path treated a dependency outage as a process failure and caused repeated restarts while the dependency remained down.

These cases matter to teams running extension APIs, controllers that watch large collections of objects, and components in the request path that depend on deep dependency checks.

1. Extension API servers: how a dependency update broke unrelated clients

In one of the clusters, Helm deployments began failing for charts that contained only standard Kubernetes resources. The charts had nothing to do with the extension API that caused the failure. Users who had never heard of that extension saw validation errors before their requests reached the API server. The failure path ran through metadata, via a schema document that combined definitions from the APIs registered in the cluster.

To understand why, you need to know how Kubernetes exposes API metadata to clients. Kubernetes can be extended through aggregated API servers, which add new APIs without modifying the main API server. To users, an aggregated API looks native: it has a group, version, kind, and resource paths, and kubectl accesses it through the cluster's normal endpoint. An extension API server generates an OpenAPI schema that describes its objects and fields. Clients retrieve this schema to understand and validate resources before sending requests.

OpenAPI v2 and v3: Kubernetes exposes this schema contract in two forms, and the difference determines the blast radius of a broken extension. OpenAPI v2 is a single combined document available at /openapi/v2, which contains definitions from the main API server and from aggregated extension APIs. OpenAPI v3 separates schemas by API group and version. That separation reduces impact because clients using the main API don't need to retrieve every extension's document. The v2 format creates a shared failure domain: an invalid definition from one extension can prevent clients from parsing the document used to validate unrelated resources.

Explicit OpenAPI model names: Kubernetes PR #131755 made OpenAPI model names explicit. Before this change, Kubernetes could derive a model name from a Go package path. That made it easy to start with a valid schema, but it also tied published names to implementation details. Moving code or reorganizing packages could change a name that OpenAPI clients saw. The more recent approach lets API authors declare a stable model package, and code generation uses that declared value when building schema definitions and references.

Before the change, a schema could expose a definition derived from the raw Go package path. After the change, once the package declares an OpenAPI model namespace, the same definition uses an explicit, stable name. A missing declaration: an extension API server updated the dependency on Kubernetes API machinery without including the +k8s:openapi-model-package annotation required by the new model naming generation. Its generated code continued to provide raw Go package paths when building schema definitions. The previous API machinery transformed those paths into OpenAPI-safe names; the more recent behavior expects the extension to provide a declared model package. Without that declaration, the extension began publishing raw package paths as OpenAPI model identifiers. Because those paths contain /, they made the resulting schema invalid. The extension server started correctly and continued serving its API. Process health checks don't validate the OpenAPI document a process publishes. The problem surfaced in clients reading OpenAPI v2. Because v2 combines API schemas into a single document, an invalid definition from one extension can cause a client to report an error before sending the requested API call, even when the requested resource belongs to another API. Helm's behavior made this especially visible. Depending on the Helm version, command, and validation configuration, Helm retrieves the OpenAPI v2 document before applying a chart. A chart that uses only ordinary Kubernetes resources can fail because an unrelated aggregated API publishes an invalid schema. Clients using v3 for core resources follow a narrower path.

What it means for an SMB and what we recommend

If you manage a Kubernetes cluster, even a small one, these failures can affect your deployments and the availability of your services. The good news is that they can be mitigated with concrete practices. At ForgeNEX we recommend:

• Review your extension API servers: if you use aggregated APIs, make sure they explicitly declare the OpenAPI model package (+k8s:openapi-model-package) and that they generate valid schemas. It's not enough for the process to be alive; validate the OpenAPI schema it publishes.
• Prioritize OpenAPI v3: when possible, use clients and tools that consume OpenAPI v3 to reduce the shared failure domain. If you depend on v2, keep in mind that an error in one extension can block unrelated operations.
• Control controller startup: avoid massive, synchronized restarts of controllers that rebuild local views (informer). Stagger deployments and consider rate limits to protect the API server.
• Design appropriate health checks: don't turn a dependency outage into a process failure with overly deep liveness probes. Use readiness for dependencies and avoid cascading restarts.
• Monitor the critical path: observe not only process health, but also the validity of the metadata they publish and the impact on clients.

In an SMB environment, where teams are small and resources limited, these details make the difference between a controlled incident and a prolonged outage. If you need help reviewing your Kubernetes configuration, at ForgeNEX we can help you. You can also read our articles on KubeVirt and the future of VMs in Kubernetes or on AI agents that escape to stay up to date on infrastructure trends.

Source: The New Stack. Analysis and adaptation: ForgeNEX.