Keeping a shared Slurm cluster for foundation model development in sync entails keeping several separately maintained pieces of configuration matched to what’s running on the nodes. The cluster’s shape lives on multiple configuration files, its software in custom scripts, its network topology in files that must be edited as nodes change, and its users in a roster that an administrator must maintain and rerun. Over time these pieces drift away from what’s actually running, and nothing tells you when they do.
We believe cluster configurations should be managed the way you manage the rest of your AWS infrastructure. You declare what you want through an API or the console, AWS validates it before it’s applied, the cluster is kept in that state continuously, and you can see whether it’s working. During 2026, Amazon SageMaker HyperPod brought that approach to Slurm clusters, with API-driven Slurm configuration, AMI-based configuration, continuous provisioning, automatic topology management, and deep health checks. Today, we launched cluster extensions, which bring the same approach to the software and settings on running nodes, with managed extensions for observability and user management. However, through all of it, you still keep full access to every node in your cluster, including the controller node. You can SSH in, configure nodes directly, and run your own lifecycle scripts. The APIs give you less to maintain, direct access gives you more control, and you choose the balance.
In this post, we walk through the features we introduced in 2026 and explain why the software on running nodes needed a different mechanism.
Source link







