AI Agents and Kubernetes: Who Reasons and Who Executes

In Kubernetes we have been living with "agents" for years, even if they are not the ones now starring in all the conversations about artificial intelligence. A controller observes the current state, compares it with the declared desired state, and acts to close the gap between the two. The temptation to say that an AI agent and a Kubernetes operator are the same thing is high. They are not, and the difference matters a great deal when we are talking about a production cluster.

The question is no longer theoretical. In the last year and a half, agent tools have gone from being an experiment to forming part of the cloud native stack. The CNCF has accepted agent frameworks into its Sandbox, and the Kubernetes SIG Apps has incorporated Agent Sandbox as a subproject to run agent workloads securely. Many platform teams are already deciding what an agent can touch in a live cluster, usually through a pull request every time and without a written policy in place. That is, the boundary between reasoning and executing is being defined by convention, not by design.

Two different ways of understanding control

A Kubernetes operator is designed to be predictable: it reconciles resources according to coded rules, retries failed operations, and converges toward the declared state. An AI agent is useful when the desired outcome is clear but the path is not. It can investigate an unknown situation, gather context from various systems, weigh alternatives, and propose a course of action.

The most sensible architectural pattern combines both strengths: the agent provides reasoning and context; operators, policy engines, and the rest of the deterministic systems retain control of execution. It is a classic separation of responsibilities, applied to a new component.

kagent: a concrete example

The open source project kagent, within the CNCF Sandbox, illustrates this pattern well. It runs AI agents as Kubernetes resources and connects them with Kubernetes, Helm, observability, and its own tools. An agent can inspect resources, retrieve logs and events, query metrics, review documentation, and coordinate specialized agents.

That does not mean that all tools should be able to run freely. Non-sensitive read operations can be automated, while changes such as applying, patching, or deleting resources may require explicit approval. This way the agent investigates broadly without an uncertain reasoning process having unlimited authority over the cluster.

The resulting operational loop would be: observe, diagnose, propose, approve, reconcile, and verify. The first three steps benefit from the agent's reasoning; reconciliation is handled by Kubernetes with its operators; verification closes the loop without taking for granted that an accepted change has produced the intended effect.

Karpenter: the operator is not replaced, it is supervised

Let's think about a cluster where Karpenter provisions capacity correctly, but the team still sees underutilized nodes, slow consolidation, Spot instance rotation, or pending pods. Karpenter is already an operator: it evaluates scheduling constraints and reconciles NodePools and NodeClaims. Replacing that logic with a language model would make a control loop that works less predictable.

The sensible thing is to place an operations agent above Karpenter. It can review the NodePool and EC2NodeClass configuration, pending pods, Kubernetes events, Pod Disruption Budgets, Karpenter logs, and Prometheus metrics, and ask itself why the observed behavior differs from the team's goal. The investigation could discover that a Pod Disruption Budget is blocking consolidation, that a do-not-disrupt annotation is keeping a node fixed, or that consolidateAfter is too aggressive for a workload with spikes.

The agent's output should not be a vague recommendation like "optimize the NodePool." It should present the evidence, identify the trade-off, and generate a scoped configuration patch: how many nodes could be affected, for how long stability will be observed, or what constraints are preserved when expanding instance options. Then the patch goes through an approval or policy gate, and Karpenter reconciles with its usual logic.

Argo CD applies; the agent builds the case

Another common scenario: a platform team updates a Helm chart managed by Argo CD. Changing the version in Git is simple; understanding how it will affect the running application requires more work. Argo CD shows which resources differ; an agent can explain what those differences mean. For example, a new version of the chart could rename a value used to configure the readiness probe. The agent compares release notes, existing values, and rendered manifests to detect whether the update would silently fall back to a default value.

The result should be a focused pull request that updates the version and the affected values, explains the operational risk, and defines pre-deployment checks. Once reviewed and merged, Argo CD renders and syncs. The agent builds the case; Argo CD applies.

What it means for an SME or an IT team

If you manage infrastructure with Kubernetes, even modestly, this discussion affects you sooner than it seems. Some practical recommendations:

• Do not delegate execution to the agent. Let it propose, not apply. Changes to the cluster must go through existing operators, pipelines, or policies.
• Define in writing what it can touch. If today the boundary is a convention, turn it into policy: automatic reads, writes with approval.
• Demand evidence and scope. A good agent does not say "optimize"; it explains the problem, the trade-off, and the maximum impact of the change.
• Verify afterward. The fact that a change is accepted does not mean it has worked. Close the loop with metrics and acceptance criteria.
• Start with the boring stuff. Diagnosis of pending pods, log review, or chart comparison before touching capacity or networking.

This approach fits with what we have already discussed on the blog about the three bottlenecks when scaling Kubernetes and with the increasingly widespread idea that AI risk is no longer measured asset by asset. AI provides context and speed of analysis; infrastructure still needs determinism to execute.

At ForgeNEX we see it this way: the agent is a good analyst, not an operator with the keys to the cluster. Designing that separation from the start is cheaper than discovering it after an incident.

Source: The New Stack. Analysis and adaptation: ForgeNEX.