Kubernetes Control Planes Break in Three Different Ways—and Need Three Different Fixes
Production clusters hit scaling walls through node churn, object state, or write velocity—not just node count. Engineers are discovering that the standard playbook for Kubernetes scale misses the real bottlenecks.
Kubernetes scaling problems don't all look the same. A training fleet that jumped from a few thousand nodes to fifteen thousand watches its kube-apiserver memory spike and etcd strain under the load. An agent platform running at modest node count but churning through thousands of session lifecycle events per minute faces a completely different crisis. A platform cluster with two hundred operators and tens of thousands of mostly-dormant objects creates yet another kind of pressure, one where every apiserver replica carries the full weight of the entire state in memory. Each cluster is maxing out in a different direction, and solving one problem does nothing for the others.
The conventional wisdom—add more apiservers—makes things worse. Each new apiserver replicates the entire watch cache, trading request capacity for memory you don't have. Yet production clusters well past 5,000 nodes are running today on unmodified upstream Kubernetes. They got there not through custom forks but through deliberate choices about what each component stores and processes.
The scaling itself isn't the hard part. What's missing is the assembly instructions.
Three kinds of scale
When people ask how large a control plane needs to be, the answer almost always comes back in node count. That metric leaves the second and third clusters tuned for problems they don't actually face.
Node-bound. Training and reinforcement-learning fleets are large and homogeneous. The pressure comes from every kubelet reporting in. At 100,000 nodes with a 10-second heartbeat, liveness checks alone generate roughly 10,000 writes per second, with lease renewals accounting for approximately 70% of write throughput. The control plane spends most of its effort on nodes continuously announcing their presence.
Churn-bound. Agent platforms are smaller but far more intense. Many agent sessions run on a modest pool of worker pods, with each actor's lifecycle expressed as a custom resource. The busiest and shortest-lived objects are CRDs, and write rates have nothing to do with machine count.
State-bound. Platform clusters contain many object types, most of them inactive. The cost isn't writes but resident bytes multiplied across every apiserver holding a complete copy. Adding apiservers directly amplifies this problem.
These three axes are independent. You can saturate any one while keeping the other two well below capacity. A tuning strategy that only knows node count will happily optimize for the wrong constraint.
Seams and recipes
Seams are the contract points where Kubernetes components connect—usually at network boundaries and designed to be substitutable. You can replace what's behind a seam without patching upstream code.
Think of them the way a tailor thinks about seams in a jacket. Cut somewhere that isn't a seam, and you've made a very expensive rag.
Recipes are scale-tested configurations built around those seams: executable configurations that either pass their scale test or fail it, with continuous integration confirming the day they stop working.
The four seams
Seam 1: the storage contract. Which store holds which resource types. This relieves churn pressure. Not all cluster state requires the same durability guarantees. Events and node leases are ephemeral, leader-election leases are ephemeral and high-traffic, and none need the same durable, Raft-replicated store as Deployments. KEP-4743 defines the storage-interface boundary, and –etcd-servers-overrides routes resource types to different backends today. etcd isn't the bottleneck, and nothing here argues for replacing it; a general-purpose consistent store shouldn't absorb traffic when no business logic requires durability.
The gap: –etcd-servers-overrides only works with built-in resources, so a churn-bound platform can't route the custom resources generating its write storm. An enhancement in a future release should address this.
Seam 2: the node liveness contract. How liveness information reaches the control plane. This relieves node-count pressure. Two fixes compose here. First, absorb the heartbeat load: a node-facing tier in front of the control plane handles renewals and propagates only real state transitions. Second, make it cheaper: route renewals through etcd's in-memory lease refresh instead of a full Raft commit. That change lands in etcd itself, so it helps every cluster, including three-node ones.
Seam 3: the shard axis. The field that controllers, caches, and stores agree to shard on. This relieves all three pressure points. Horizontal scaling requires every layer to agree on how objects are divided. Pick the wrong axis and controllers spend their time reaching for objects in someone else's shard. KEP-5866 introduces hash-range filtering for API traffic as the foundation. Coordination, resharding, and sharding the controller manager remain out of scope and all need work.
Seam 4: cache residency. What an apiserver actually keeps in memory. This relieves state pressure. The apiserver that replicates the whole watch cache is this seam, and it's the state-bound cluster's entire problem. An apiserver that only talks to the scheduler doesn't need every object resident, and a two-shard prototype cuts heap by roughly half per shard. There's a floor: with every disableable cache off, an apiserver still carries about 0.9 KB per RBAC object, because the authorizer's informers hold them regardless.
The gap: residency doesn't exist upstream yet. KEP-5866 filters egress, so the cache still stores everything. Prototype branches and a draft KEP update are up for review.
The recipe ladder
- Separate machine pools for API serving, scheduler, KCM, etcd
- Role-Split plus a node-facing tier that absorbs the leases
- Node Plane plus resource types split across independent stores
- K shard units, each a controller plus apiserver pinned to a hash range
That last rung is on the ladder on purpose: publishing the step nobody has built is how it stays useful. The bigger gap is the index. Choices happen by node count, the one axis of three with measurements behind it. No one, including the authors, has published a scale-tested recipe for a churn-bound or state-bound cluster.
None of this is a distribution: stock upstream binaries, shipped as CI-tested configuration, so a release that changes the calculus shows up as a test failure first.
Where you can push
- Per-CRD storage configuration, for Seam 1. No KEP yet so that the use cases will shape it. If your hot objects are custom resources, bring your write rates to SIG API Machinery. Numbers off a real churn-bound cluster are the most useful thing anyone could contribute.
- KEP-5866 and the residency extension, for Seams 3 and 4. Prototype branches are up for review, and the design is not settled.
- Recipes nobody has published. Above 20,000 nodes, or serious churn, or a cluster that's simply heavy: you know something the rest of us don't.
Training fleets, agent platforms, and the unglamorous platform cluster are running out of room in three different directions, and all three should get there with stock upstream components. Most of the pieces are already on the workbench. Let's put them together.