Anyscale outlines production KubeRay practices for logs, GPU health, autoscaler tuning and head-node resilience, with Connect in private preview.
Anyscale’s latest KubeRay field guide centers on what changes when Ray workloads move from a single successful job to shared production infrastructure. The post, published Oct. 26, 2026, says the most common failures emerge around ephemeral clusters, GPU contention, autoscaler defaults and head-node disruptions, rather than in the basic RayJob flow itself.
The article describes KubeRay as the bridge between Ray and Kubernetes, and says a production RayJob typically adds GPUs, Kueue-based queueing, head-node isolation and worker scale-to-zero behavior. It also cites a public Ray Summit 2024 talk in which Ray Train scaled to 10,000 Kubernetes nodes, using that example to frame the gap between quickstart deployments and fleet-scale operation.
Several concrete operational scenarios are identified. First, logs and dashboards can disappear with short-lived RayJobs, which makes failures in the first minutes of a run difficult to diagnose. Second, hardware problems such as PCIe, NVLink or NFS issues can surface as application errors even when a pod still reports Ready. Third, default autoscaler settings can break down under contention, for example when placement groups form slowly or retries leave resources stranded. The post also says head-node drains, memory pressure and shared-cluster side effects from service meshes or other platform controls can take down workloads across the cluster.
To address those cases, the post recommends six practices: persist logs and events to object storage or an external log stack before the cluster is deleted; treat GPU health as a scheduling signal using DCGM exporter alerts and node health checks; tune for production contention rather than quiet-cluster behavior; design for head-node loss by keeping the head free of task scheduling and using fault tolerance where needed; fit Ray into existing namespaces, policies and meshes; and alert on failures that matter while automating the first response where possible.
Source link
