Google donates AI agent tools to CNCF, targeting idle compute costs on Kubernetes

Kubernetes has emerged as the standard infrastructure for distributed model training and inference work, yet a new challenge has surfaced: the financial impact of running AI agents that remain idle for extended periods. When containers or virtual machines hosting these agents stay provisioned and waiting, organizations incur charges regardless of whether active computation is occurring. To tackle this problem and stabilize the cost profile of agent-based workloads, Google LLC has contributed three initiatives to the Cloud Native Computing Foundation, according to Allan Naim, director of product management for AI infrastructure at Google Cloud.

The shift reflects a fundamental change in how Kubernetes is being used. "Act 2 is a few years back; Kubernetes became the de facto for inference and training," Naim said. "What's really interesting right now is we're embarking on this Act 3 … where the typical patterns are very, very different from how Kubernetes was designed."

Open building blocks for distributed training and inference meet agentic AI

Idle time represents a substantial economic drain in agentic workloads. Agents typically remain inactive between 90% and 95% of the time while waiting for model responses, test execution or human decision-making, Naim explained. To address this, Google developed Agent Substrate, an open-source runtime environment that isolates AI agents within Kubernetes. The system captures a memory snapshot and file state when an agent pauses, then deallocates its compute resources. Upon resumption, the runtime restores the agent from its saved state in an available sandbox environment.

The efficiency gains are substantial. "It helps from an economic standpoint," Naim said. "We've seen 10x reduction in cost in terms of running Agent Substrate on Kubernetes versus running these agents directly on dedicated virtual machines and containers."

Beyond cost management, security poses another critical consideration. Google open-sourced gVisor in 2018, a security layer that operates above the host kernel to intercept and either block or log every system call made by an agent, according to Gari Singh, product manager at Google Cloud.

"Agents and agent harnesses are designed to solve a problem — and to try to solve a problem — until they can't solve the problem," Singh said. "They will find all kinds of clever workarounds. They know everything that's around. You want to make sure that it didn't go make some low-level change to [your] low-level security, didn't make an outbound network call, didn't try to open a file that it shouldn't."

Google is collaborating with Anyscale Inc. on KubeRay, which enables Anyscale's Ray platform to operate natively within Kubernetes, allowing platform teams to provide distributed training and inference capabilities across multiple research groups, Singh noted. The third contribution, OpenRL, extends this model to reinforcement learning, a technique used to establish safety guidelines for fine-tuned models.

https://www.youtube.com/embed/wv2C9Gf8aAs?feature=oembed

Reinforcement learning has traditionally required significant specialized expertise. "In the past, to do reinforcement learning, you needed to hire a dozen research scientists with PhDs," Naim said. "Our goal was, 'How can we bring reinforcement learning into the world of platform engineering so that you can run reinforcement learning on your Kubernetes environment and take advantage of it to actually do continuous improvement for your agentic workload?'"