Reasoning and Execution: How AI Agents and Kubernetes Operators Work Together
AI agents excel at reasoning through complex situations, while Kubernetes operators provide deterministic control. The safest approach combines both, letting agents investigate and propose while operators execute approved changes.
Kubernetes has long depended on agents—though not the reasoning systems now dominating industry conversations. A Kubernetes controller observes the current state, measures it against a desired state, and executes actions to eliminate the gap. This structural similarity to modern AI agents masks a fundamental difference: they serve distinct purposes and should not be treated as interchangeable.
The question has moved from academic to practical. Agent tooling has graduated from experimental side project to established component of the cloud-native ecosystem over the last 18 months. The CNCF has brought agent frameworks into its Sandbox program, and Kubernetes SIG Apps has established Agent Sandbox as a subproject dedicated to running agent workloads with appropriate safeguards.
Many platform teams are already determining what permissions agents should have within production clusters, frequently on a pull-request basis and without formal governance structures. Currently, the separation between reasoning and execution relies on informal convention rather than deliberate architectural design.
Operators Execute; Agents Reason
Kubernetes operators are engineered for predictability. They apply encoded rules to reconcile resources, retry operations that fail, and drive systems toward their declared target state. AI agents deliver value in scenarios where the objective is known but the method to reach it remains unclear. They can examine unfamiliar conditions, pull context from multiple sources, evaluate competing priorities, and suggest next steps.
A more secure design pattern leverages both capabilities. The agent furnishes reasoning and situational awareness. Operators, policy mechanisms, and other rule-based systems maintain authority over what actually changes.
A Reasoning Layer for Kubernetes
kagent, a CNCF Sandbox initiative available as open source, demonstrates this pattern in practice. It deploys AI agents as Kubernetes resources and integrates them with Kubernetes itself, Helm, monitoring systems, and proprietary tools. An agent running in this framework can examine resource definitions, pull logs and events, access metric data, review documentation, and orchestrate other specialized agents.
Not all operations should be freely available to an agent. Read-only queries that pose no risk can execute without intervention, while mutations—applying, modifying, or removing resources—should demand human sign-off. This approach permits the agent to conduct thorough investigation while preventing an imperfect reasoning process from making uncontrolled changes to the cluster.
The operational sequence becomes: observe, diagnose, propose, approve, reconcile, and verify. Agentic reasoning strengthens the first three phases. Kubernetes and its operators own reconciliation. Verification completes the cycle by confirming that an accepted change actually delivered the intended outcome.
Karpenter: A Case Study
Imagine a cluster where Karpenter manages capacity provisioning correctly, yet the operations team observes unused node capacity, sluggish node consolidation, excessive Spot instance churn, or pods stuck in pending state. Karpenter functions as an operator itself—it analyzes scheduling requirements and manages NodePools and NodeClaims. Replacing this logic with an LLM would undermine the reliability of the control loop.
An operations agent built with kagent should sit above Karpenter, not replace it. It can review NodePool and EC2NodeClass settings (in EKS environments), examine pending workloads, scan Kubernetes events, inspect Pod Disruption Budgets, review Karpenter logs, and pull relevant Prometheus data. From this evidence, it can reason about why actual behavior diverges from the team's objectives.
The investigation might reveal that a Pod Disruption Budget is preventing consolidation, a do-not-disrupt label is anchoring a node, or an eviction policy is blocking node removal. It might discover that consolidateAfter settings are too aggressive for workloads with traffic spikes, generating unnecessary disruption. It could also spot instance type constraints that leave Karpenter with insufficient options when selecting Spot resources.
The agent's recommendation should be precise, not vague. Rather than suggesting "optimize the NodePool," it should present the supporting evidence, articulate the tradeoff, and generate a specific configuration change. A modification to disruption settings should quantify the maximum number of nodes at risk. A consolidation adjustment should specify how long stability will be monitored. An increase in instance variety should respect architectural, zone, and workload constraints.
The proposed patch enters an approval or policy review stage. After approval, Karpenter applies its existing scheduling and disruption algorithms to reconcile. The agent then monitors the outcome over a set timeframe to verify whether consolidation improved without introducing pending-pod delays or unwanted disruption. If results fall short of acceptance criteria, the system can recommend or execute a rollback using the same controlled mechanism.
Argo CD Deploys; The Agent Explains
Consider a platform team updating a Helm chart that Argo CD manages. Modifying the chart version in Git is simple. Predicting how that change will influence a running application demands deeper analysis. Argo CD can display which resources will change; an agent can interpret what those changes mean for the application's behavior.
Suppose a new chart version renames a configuration parameter that the team uses to set a readiness probe. The agent can cross-reference release notes, current values, and the rendered manifest to determine whether the upgrade would silently revert to a default setting. It should convert this finding into a targeted pull request that updates the chart version, modifies the affected values, documents the operational risk, and specifies validation steps before deployment.
Once the pull request is reviewed and merged, Argo CD renders the chart and applies the resulting manifests according to the application's sync configuration. The agent then examines Argo CD's sync and health status alongside application performance metrics. A successful sync does not guarantee that the upgrade maintained latency, error rates, or uptime.
Should those checks fail, the agent can suggest a Git revert or a recovery strategy through the same approval workflow, taking into account any data or schema modifications that a simple revert cannot address. Responsibility remains divided: the agent investigates, proposes, and validates; Argo CD reconciles the approved desired state.
Autonomy Must Be Earned
Organizations can adopt this model incrementally. Begin with an agent that only reads. Advance to allowing the agent to generate patches or pull requests. Progress to permitting approved, low-risk modifications with explicit boundaries and rollback procedures. Reserve full autonomy for actions that are well-understood and produce measurable results.
The important boundary is not whether an agent can call kubectl, Helm, or a cloud API. It is whether every action is authorized, bounded, observable, and reversible.
Kubernetes can host agents that help manage Kubernetes, provided each component maintains the role it performs best. Agents reason about uncertain scenarios, operators reconcile declared state, and policy enforces constraints on both. Humans bear responsibility for determining how rapidly the system can transition from recommendation to autonomous action.