How Kubernetes Control Planes Fail: Three Hidden Amplification Paths

Kubernetes control planes are engineered to isolate individual component failures, yet their resilience ultimately hinges on how critical-path components perform under load. At large fleet scale, a component need not fail outright to destabilize the entire system. A properly functioning mechanism can magnify a localized defect by way of shared metadata, coordinated startup procedures, or overly sensitive health checks.

Consider the informer pattern, which supplies controllers with a local snapshot of cluster state, enabling workers to make forward progress without hammering the API server with repeated queries. This design works well when controllers start independently. But when many controllers restart at once across a fleet, the simultaneous reconstruction of those snapshots can overwhelm the control plane. The informer mechanism itself becomes the failure amplifier.

At fleet scale, a component does not have to crash to contribute to an outage. A valid mechanism can amplify a local defect through shared metadata, synchronized startup work, or an overly aggressive health check.

This examination covers three concrete scenarios. First, an extension API server released an invalid schema following a dependency update, blocking clients that had never interacted with its API. Second, DaemonSet agents restarted en masse across a large fleet, converting routine initialization into a request deluge the API server could not handle. Third, a component in the API request path performed a deep dependency check as a liveness probe, treating a transient outage as a process failure and cycling through restarts while the dependency remained down.

These scenarios are relevant to teams deploying extension APIs, controllers that observe large object collections, and components in the API request path that depend on deep dependency validation.

Extension API Servers: How a Dependency Upgrade Broke Unrelated Clients

On one cluster, Helm chart deployments began failing for charts containing only standard Kubernetes resources. These charts had no connection to the extension API responsible for the failure. Users unfamiliar with the extension encountered validation errors before their requests even reached the API server. The failure path traveled through metadata—specifically, a schema document that merged definitions from all APIs registered in the cluster.

Understanding this requires examining how Kubernetes publishes API metadata to clients. Kubernetes supports extension through aggregated API servers, which introduce new APIs without altering the core API server code. To end users, an aggregated API appears indistinguishable from a native one: it has a group, version, kind, and resource paths, and kubectl reaches it via the cluster's standard API endpoint. An extension API server produces an OpenAPI schema documenting its objects and fields. Clients fetch this schema to understand and validate resources before sending requests.

OpenAPI v2 and v3

Kubernetes exposes this schema contract in two formats, and the distinction shapes the scope of impact from a broken extension. OpenAPI v2 is a single consolidated document served at /openapi/v2, combining definitions from the core API server and all aggregated extension APIs. OpenAPI v3 organizes schemas by API group and version. This separation limits the blast radius because clients using a core API do not need to pull every extension's document.

The v2 format creates a shared failure domain: an invalid definition from one extension can prevent clients from parsing the document used to validate unrelated resources.

Explicit OpenAPI model names

Kubernetes PR #131755 introduced explicit OpenAPI model names. Previously, Kubernetes could infer a model name from a Go package path. This approach allowed starting with a valid schema, but it also bound published model names to implementation specifics. Relocating code or restructuring packages could alter a name that OpenAPI clients observed. The newer approach permits API authors to specify a stable model package, and code generation applies that declared value when constructing schema definitions and references.

Before the change, a schema could expose a definition derived from the raw Go package path:

{
   "com.example.platform.pkg.apis.metrics.v1.MetricReportSpec": {
     "type": "object"
   }
}

After the change, once the package declares an OpenAPI model namespace, the same definition uses an explicit, stable name:

{
   "com.example.platform.metrics.v1.MetricReportSpec": {
     "type": "object"
   }
}

A missing declaration

One extension API server upgraded the Kubernetes API machinery dependency but omitted the +k8s:openapi-model-package annotation required by the updated model name generation logic. Its generated code continued supplying raw Go package paths when assembling schema definitions. Earlier API machinery converted those paths into OpenAPI-safe names; the newer behavior expects the extension to furnish a declared model package. Without that declaration, the extension began publishing raw package paths as OpenAPI model identifiers. Since those paths contain /, they rendered the resulting schema invalid.

The extension server started successfully and kept serving its API. Process health checks do not validate the OpenAPI document a process publishes. The problem surfaced in clients reading OpenAPI v2. Because v2 merges API schemas into one document, an invalid definition from one extension can cause a client to report an error before dispatching the requested API call, even when the requested resource belongs to a different API.

Helm's behavior made this particularly apparent. Depending on the Helm version, command, and validation configuration, Helm retrieves the OpenAPI v2 document before deploying a chart. A chart using only standard Kubernetes resources can fail because an unrelated aggregated API publishes an invalid schema. Clients using v3 for core resources follow a narrower path: they retrieve the relevant core API document rather than the extension schema containing the problem.

Test the schema you publish

The most instructive test validates the artifact that clients actually receive. Following an upgrade, start the extension and retrieve both schema formats:

kubectl get --raw /openapi/v2 > openapi-v2.json
kubectl get --raw /openapi/v3/apis// > openapi-v3.json

Parse and validate those documents before deploying the extension to production. Also test with clients that validate requests against the OpenAPI v2 schema. These tests follow the user path and surface problems that endpoint health checks cannot catch.

When a Rolling Restart Becomes a Request Storm

The cost of rebuilding a local view

A DaemonSet runs a single Pod replica on every node in a cluster. It is well-suited for monitoring, network policy, logging, and security agents that require an up-to-date view of cluster Pods. For instance, a network agent may need to correlate an IP address with its owning Pod for policy enforcement or telemetry.

The standard approach to maintaining that view is through an informer: a client mechanism that retrieves the required objects and then watches for changes to keep a local cache synchronized.

On a cluster with several thousand nodes, a cluster-wide Pod informer translates to several thousand potential initial list operations. When agents start at staggered intervals, the load distributes across the API server's capacity without disruption.

When recovery synchronizes the work

The calculation shifts when recovery synchronizes the work. In one incident, a DaemonSet ran across thousands of nodes, and its agents crashed repeatedly across much of the fleet. An operator performed a rolling restart to restore them. The DaemonSet's maxUnavailable setting appeared protective because it restricts how many Pods an update can make unavailable simultaneously.

Those Pods were already unavailable. Their replacements did not necessarily face the same pacing constraints. Many agents started nearly at the same time.

Each new agent instantiates its informer. Each informer begins by listing the same Pod collection. A recovery operation intended to shrink the failure footprint can therefore generate a large, synchronized wave of requests against the API server.

The cost of a list request

Informers typically issue their initial request with the resourceVersion parameter set to 0. By using 0, the client permits the API server to respond from its watch cache in memory rather than executing a direct read from etcd, the API server's backend datastore. This is ordinarily the fast path. It sidesteps repeated storage reads and leverages a view of the state the API server has already prepared for watches.

For many years, that path carried an unintuitive constraint. The watch cache could not supply pagination in the same manner that lists served from etcd could. A client could specify limit=500 and reasonably assume it had capped the response to 500 objects. In certain request paths, when the list was served from the watch cache with resourceVersion=0, the API server could disregard the limit and return all matching objects.

On a large cluster, a response containing all Pods may encompass tens of thousands of objects and consume hundreds of megabytes as the API server decodes, filters, serializes, and transmits it. When thousands of replacement agents request the same full collection in parallel, the server can deplete available memory and CPU constructing many copies of the same response. Regular control plane traffic then gets queued.

Reducing the startup load

The Kubernetes community has enhanced this portion of the stack. API Priority and Fairness list-cost estimation was corrected in Kubernetes 1.31. WatchList can also substitute for the traditional list-then-watch startup sequence in clients that support it. These enhancements lower the per-request cost, but they do not alter the underlying design trade-off: a cluster-wide informer in a DaemonSet still multiplies its startup cost by the node count, and a retry policy can still synchronize that cost during a failure.

The key question for an agent owner is: what is the smallest Kubernetes resource collection this informer can watch and still accomplish its purpose?

An agent that only needs Pods in one namespace should not list all Pods across the cluster. An agent that needs Pod metadata should consider a metadata-only informer rather than transferring complete Pod specifications. An agent interested in service backends may be able to use EndpointSlices instead of listing every Pod because EndpointSlices describe service endpoints directly and are typically much smaller than the full Pod inventory.

An agent that genuinely needs Pod-to-IP ownership, labels, or other Pod metadata may still need to watch Pods, but it should confine that watch to the relevant namespaces, nodes, or labels whenever its semantics permit. The aim is not to eliminate Pod informers. It is to avoid reconstructing more cluster state than the agent can use, particularly when recovery causes many instances to start together.

When a Liveness Check Restarts What It Cannot Fix

A liveness probe that checks downstream dependencies will restart a process whenever those dependencies are unreachable. If the process sits in the API request path, each restart can disrupt API traffic. When the dependency outage persists, the probe drives a loop of restart, initialize, fail dependency check, and restart again. Each cycle adds latency without fixing the dependency.

The probe may faithfully execute its contract. The contract may still be wrong.

Static stability depends on cached key material

Kubernetes control planes may rely on an external Key Management Service, or KMS, to encrypt Secrets. The KMS provider uses that service for key operations, while the API server can hold cached encryption material in memory. That cached state is part of the system's availability design. A temporary loss of KMS connectivity may postpone operations that require fresh key material without immediately rendering the API server unable to serve useful requests.

A Liveness probe that tested the wrong thing

In one incident, an encryption provider that bridged Kubernetes with an external KMS had a liveness probe monitored by the kubelet. The probe performed a fresh encryption operation, followed by a decryption operation, against the external KMS every few seconds. When a network fault made the KMS endpoint unreachable, the probe failed, and the kubelet restarted the provider. The restarted provider could not establish a new KMS session, so it kept restarting while the dependency remained unreachable.

The probe accurately signaled that the KMS endpoint was unreachable. It could not determine whether the provider process could no longer serve its local interface, or whether the API server had consumed the cached encryption material that let it continue working. Neither conclusion followed logically from a failed call to the external KMS.

The cascade through API server health

The risk extended further because the API server's composite /healthz check incorporated the KMS provider's status. A failed provider could therefore displace an API server with usable cached encryption state. The replacement API server had no warm encryption cache, needed to establish a new KMS session, and faced the same unreachable dependency. Repeated replacements could convert a transient KMS impairment into a loss of control plane serving capacity.

Preserve capability during an impairment

This illustrates the distinction between testing dependencies and testing capabilities. KMS reachability was a useful monitoring signal, but it was not sufficient proof that either the provider process or the API server had forfeited all useful capability. The liveness probe should ascertain whether the provider is stuck or unable to serve its local interface. A separate dependency check can notify operators that KMS is unavailable without automatically triggering a restart.

The same reasoning applies to any component in the control plane path. A cached credential, an established connection, or an in-memory view of state may be the only thing permitting useful work to continue while an upstream service recovers. This behavior is already embedded in components such as kube-proxy and CoreDNS: kube-proxy continues using its last-known Service and Endpoint state, while CoreDNS continues answering from the Kubernetes data it already has, even when the API server is temporarily unavailable.

Each case shares a structural property: a mechanism can operate correctly within its own contract yet still violate the availability contract of the system it belongs to. Component-level correctness is necessary but insufficient.

Restarting every process that cannot reach every dependency discards cached state, increases the amount of state that must be rebuilt, and creates synchronized demand on an already impaired dependency.

Each case exhibits a structural property: a mechanism can operate correctly within its own contract yet still breach the availability contract of the system it belongs to. Component-level correctness is necessary but insufficient for system availability.

The overarching design principle is to examine every mechanism on the critical path for its behavior when its assumptions are violated. Schema aggregation assumes extensions publish valid schemas. Rolling restart pacing assumes Pods are available to become unavailable. Liveness probes assume restart moves a process closer to health. When those assumptions fail, the mechanism can amplify the failure it was designed to contain.

For platform teams operating shared control plane components at scale, these are practical review criteria for upgrades, recovery exercises, and health-check design:

• Validate the schema artifacts that clients receive, not only the extension process.
• Limit informer scope and startup work to the smallest resource collection the component requires.
• Separate dependency monitoring from liveness decisions when a component can retain useful capability during an impairment.
• Test recovery behavior when many replicas restart simultaneously.

These practices help keep a local failure local, preserve useful capability during dependency outages, and prevent ordinary recovery work from becoming a control plane incident.