Kubernetes control planes are designed to contain individual failures, but their availability still depends on how components in the critical path behave under pressure. At fleet scale, a component does not have to crash to contribute to an outage. A valid mechanism can amplify a local defect through shared metadata, synchronized startup work, or an overly aggressive health check. 

For example, an informer gives a controller a local view of cluster state, letting its workers make progress without repeatedly querying the API server. This is efficient when controllers start independently. When many controllers restart simultaneously across a fleet, rebuilding those views can place significant pressure on the control plane. The informer mechanism itself becomes the amplifier.

“At fleet scale, a component does not have to crash to contribute to an outage. A valid mechanism can amplify a local defect through shared metadata, synchronized startup work, or an overly aggressive health check.”

This article examines three cases. In the first, an extension API server published an invalid schema after a dependency upgrade and blocked clients that had never used its API. In the second, DaemonSet agents restarted across a large fleet, turning routine startup work into a request storm the API server couldn’t absorb. In the third, a deep liveness check on a component in the API request path treated a dependency outage as a process failure and drove repeated restarts while the dependency remained unavailable.

These cases matter for teams running extension APIs, controllers that watch large object collections, and components in the API request path that rely on deep dependency checks.

Source link