AWS Lessons: How to Manage Zonal Failures in Kubernetes at Million-Cluster Scale

AWS Lessons: How to Manage Zonal Failures in Kubernetes at Million-Cluster Scale

  • 11/Jul/2026
  • ForgeNEX by ForgeNEX
  • AI

The Challenge of Zonal Failures in Kubernetes

Managing obvious failures is simple: a server goes down, you spin up another. But the failures that truly impact are zonal ones, where an entire AWS Availability Zone becomes inaccessible. AWS has operated millions of Kubernetes clusters and learned key lessons to mitigate these events.

what-running-kubernetes-across-millions-of-cluster-0.jpg

Resilience Strategies: EKS Zonal Shift

The flagship tool is EKS Zonal Shift, which allows you to move workloads out of an affected zone in a controlled manner. For SysAdmins and DevOps, this means fewer pages at 3 a.m. and more time to innovate. For the business, it translates into higher availability and less revenue loss.

what-running-kubernetes-across-millions-of-cluster-1.jpg

Impact on Daily Operations

The most valuable lesson: automate detection and response to zonal failures. Don't wait for the cluster to degrade. Implementing pod disruption budgets and topology spread constraints policies is essential. As we saw in our article on compact mainframes, resilience is not optional.

what-running-kubernetes-across-millions-of-cluster-2.jpg

Recommendations for Your Infrastructure

1. Use multiple Availability Zones from the design stage. 2. Test zonal failures regularly. 3. Monitor latency between zones. 4. Consider solutions like EKS Zonal Shift or multi-cluster alternatives. Security and best practices in Microsoft 365 also apply here: anticipate failure.


Source: The New Stack. ForgeNEX analysis.

Share: