Kubernetes in the face of the flood of AI agents: what it entails

The conversation about AI agents has stopped being a promise of the future and become an infrastructure problem. In the cloud-native ecosystem, people are now openly talking about how a single human request can trigger around twenty agent requests. And those requests also expect responses in under 200 milliseconds, while a new Kubernetes pod is only guaranteed to start in around one second. That gap between expectation and reality is what is pushing the orchestrator to evolve.

CNCF directors Bob Killen and Daniel Krook have put this on the table in an interview in Luca Berton's Udienza series. The central idea they convey is that Kubernetes is becoming the operating system of AI: the default base layer on which more specific projects are built. For an SMB, this does not mean having to set up a cluster tomorrow, but it does mean understanding where the ground on which many tools will rest is heading.

The bottleneck is no longer the model, it is the platform

Over the last two years, the conversation about AI in the enterprise has focused on the model: which provider, which context window, what price per token. However, when moving from a demo to real use, the problem shifts. Agents are not a one-off query; they are processes that call each other, query tools, wait for confirmations and start again. That pattern multiplies the load and demands a platform capable of scaling, isolating and observing.

That is why the CNCF itself is pointing to emerging projects at different layers of the stack. On the one hand, llm-d, aimed at distributed inference of language models. On the other, Agent Substrate, a runtime designed for large-scale agent deployments. These are not pieces that an SMB is going to install on its own, but they do indicate what problems providers are trying to solve: latency, distribution and control.

kagent: agents as Kubernetes resources

Another of the interesting moves is kagent, a CNCF sandbox project that allows AI agents to be run as Kubernetes custom resources. The idea is simple to describe: an agent is declared with its system prompt, its model and a list of tools, it is applied, and a controller turns it into a running pod that speaks the A2A protocol. As Antonio Berben (Solo.io) and Felipe Vicens (Telefónica) explain in the CNCF blog, the tutorial can be run on a laptop with a local model via Ollama, although it also supports providers such as OpenAI, Anthropic, Azure OpenAI or Gemini.

There is one detail worth highlighting: the authors themselves acknowledge that the smaller models had difficulty generating valid manifests. That is, the barrier is not in the infrastructure, but in the model's ability to produce correct artifacts. In addition, the builder agent is limited by a whitelist of three tools, requires human approval before applying anything and uses RBAC so that it can only write resources of type Agent. It is a good example of how security is being designed in this area: the agent proposes, the human approves and permissions constrain.

Compliance: the gap nobody wants to fill

Alongside the technical debate, there is an organizational issue that directly affects European SMBs. Matteo Bisi, from Reevo, summed it up with a blunt phrase in the CNCF blog: «Security can write the policy. Security cannot merge the Helm chart». The problem is who takes responsibility for a Kubernetes platform complying with regulations such as NIS2 or DORA. His proposal involves building «traceability chains» in which risk, legal and security define the scope and the evidence, and then concrete artifacts and recurring tasks are handed to the platform and application teams.

If your company is not clear on who signs off on platform compliance, the problem is not Kubernetes: it is the distribution of responsibilities.

This connects with a conversation we have already covered in the blog: the CISO facing AI can no longer measure risk asset by asset. With agents that call each other and platforms that change continuously, control has to be process-based, not inventory-based.

What we look at from ForgeNEX

For an SMB, the practical reading of all this is as follows. First, do not confuse infrastructure with the use case. Kubernetes is powerful, but it is not an end in itself; if your agent workload is small, you may not need it. Second, if you already have managed services or containers in production, start thinking about agent observability as something different from that of your applications: latencies, number of hops and cost per request. Third, apply the principle of least privilege to AI as well: tool whitelists, human approval and constrained permissions.

And fourth, do not leave compliance for the end. NIS2 and DORA are not an office formality; they affect how you document, who approves and what evidence you keep. If you are building on Kubernetes or are going to do so, it is worth defining now who owns that traceability chain. In a field where one human request becomes twenty, organizational clarity is worth as much as the best orchestrator.

Source: The New Stack. Analysis and adaptation: ForgeNEX.