The first is a training fleet. It sat at a few thousand nodes for months, then a new run landed, and it’s at fifteen thousand. Kube-apiserver starts eating memory. Etcd starts complaining.

The second never passes a few thousand nodes and is having a much worse day. It’s an agent platform, and its objects are alive: sessions created, suspended, resumed, and destroyed thousands of times a minute, every transition a write.

The third is neither big nor fast: a platform cluster with two hundred operators installed and tens of thousands of objects that mostly just sit there, and every apiserver holds a copy of all of them.

Two of those are AI workloads, and one isn’t, which is the least interesting part. They’re running out of room in three different directions, and fixing one does nothing for the others.

KubeCon + CloudNativeCon is taking place in Salt Lake City from November 9 to 12. Register now!

Go looking for guidance, and you find… not much. There’s plenty of advice for getting to 5,000 nodes. Past that, or sideways from it, the internet shrugs.

So you add more apiservers. It’s a documented option in kOps and the standard answer you’ll get from most people. It makes your problem worse, because every apiserver you add replicates the entire watch cache. You bought request capacity and paid for it in memory you didn’t have.

None of these is a research project. Clusters well past 5,000 nodes run in production today on stock upstream components.

Source link