Kubernetes finds new purpose managing AI agent inference at scale
As artificial intelligence workloads shift from training to deployment and agentic systems, the container orchestration platform is experiencing renewed relevance for handling massive volumes of inference requests across distributed infrastructure.
Agentic AI systems are creating an unexpected resurgence for Kubernetes, the open-source orchestration platform long prized for its stability and predictability. Jonathan Bryce, executive director of the Cloud Native Computing Foundation, explained that the shift in AI spending away from model training and toward inference serving and agent operation is driving development of new tools within the open-source ecosystem to manage surging inference request volumes. Since its inception, Kubernetes was engineered to manage dynamic, distributed workloads—a capability that has proven invaluable for rapidly scaling AI companies seeking a dependable infrastructure base.
The AI boom has sparked a massive investment and a huge build-out of infrastructure overall. Having Kubernetes built from the beginning to handle really dynamic distributed workloads as those companies were growing like crazy — they had a really reliable, solid foundation to build on at the same time.
Jonathan Bryce, executive director of the Cloud Native Computing Foundation
Bryce discussed these developments during an exclusive interview with Rebecca Knight at KubeCon + CloudNativeCon NA, broadcast through theCUBE, SiliconANGLE Media's livestreaming platform.
Scaling inference for agentic AI systems
While new models frequently dominate headlines with impressive benchmark results in mathematics and code generation, their practical value depends on high-performance serving capabilities. Early chatbot systems simplified this challenge because each user message required just a single inference call. Agentic systems operate differently, triggering multiple inference calls to complete individual tasks—a development that prompted the CNCF to establish a dedicated AI inference and agentic track at KubeCon.
We're using coding agents to say, 'Build me a system that does this,' and it'll spawn 10 sub-agents. Each of those are doing thousands of inference calls over the course of hours and just churning away.
Jonathan Bryce
The explosion in inference demand is fueling data center expansion while simultaneously raising questions about resource consumption. Two projects are working to extract greater performance from existing hardware infrastructure: the llm-d inference engine and Agent Substrate, which provides efficient hosting for long-running agents. According to Bryce, security and control represent additional obstacles, since agents may produce varying results when executing identical tasks. OpenShell, a security initiative now governed under CNCF's neutral framework following Nvidia Corp.'s contribution, achieved a significant milestone by attracting more than 300 contributing companies within months—a pace that would have previously required roughly two years.
What I am constantly thinking about is, 'How do we lower friction to collaboration?' How do we enable people to be able to move at the pace that the world and the industry is moving at, and how do we, while doing that, continue to drive that into collaboration around open solutions that can become those base and infrastructure components that we can all build on?
Jonathan Bryce
The complete video interview will be available as part of SiliconANGLE's and theCUBE's coverage from KubeCon + CloudNativeCon NA.