Multi-Node GRPO Orchestration: Ray, Slurm or Kubernetes

AI This article was created with the help of AI.

Ray vs Schedulers: The Orchestration Mismatch

When scaling Group Relative Policy Optimization (GRPO) beyond a single compute node, machine learning teams frequently debate whether to deploy Ray, Slurm, or Kubernetes. Framing this decision as a three-way architectural choice is a fundamental category error. Slurm and Kubernetes are cluster-level resource schedulers: their function is to manage physical node allocations, enforce multi-tenant quotas, and provision bare-metal compute. Ray is a distributed application runtime that operates strictly within an already provisioned resource allocation, orchestrating tasks, actors, and distributed memory objects across a dynamic cluster topology.

Ray does not replace a cluster orchestrator; it executes inside one. In production environments, reinforcement learning frameworks such as verl, OpenRLHF, and TRL do not interact directly with bare hardware. They need a cluster scheduler to allocate nodes for Ray first; Ray then expects a head-worker architecture with a single point of entry, so you start a Ray head node, start Ray worker nodes that connect to it, and run your Ray script on the head node. The actual architectural question is whether your infrastructure foundation should be Ray on Slurm or Ray on Kubernetes.

Selecting between Slurm and Kubernetes for reinforcement learning post-training requires evaluating workload synchronization, gang scheduling requirements, and interconnect topology. For infrastructure teams already evaluating the operational trade-offs of HPC versus cloud-native environments, our guides on Slurm to Kubernetes GPU migration and Kubernetes GPU node setup explore the lower-level host configurations. For multi-node GRPO specifically, the choice dictates how efficiently distributed actors synchronize weights and policy gradients.

Co-Dependent Roles: Why GRPO Demands Gang Scheduling

Unlike standard Supervised Fine-Tuning (SFT) where identical data-parallel workers execute synchronized forward and backward passes, GRPO orchestrates multiple heterogeneous, tightly coupled roles. A standard GRPO pipeline distributes compute across four distinct components: the primary policy trainer, distributed rollout workers (typically executing optimized vLLM engines), a reference model worker group to evaluate KL-divergence penalties, and a reward model or automated verifier pool.

These four roles operate in a rigid cyclic dependency. The rollout workers generate prompt completions across a sampled group, the reward and reference models score each trajectory, and the policy trainer computes loss updates and synchronizes updated weights back to the rollout engines. If any single role is starved of resources, the entire pipeline stalls. If rollout pods are running but the trainer cannot be scheduled due to resource fragmentation, the rollout workers sit idle while consuming active GPU reservations.

This dependency structure transforms gang scheduling from a performance optimization into an absolute operational requirement. Gang scheduling (also known as all-or-nothing scheduling) guarantees that all compute nodes and worker containers required for the entire GRPO pipeline are allocated simultaneously before execution starts.

• Slurm delivers gang scheduling natively: an sbatch allocation request for N nodes either reserves the full allocation block or leaves the job queued until all nodes and network fabrics are simultaneously free.
• Default Kubernetes schedulers evaluate pods independently, binding containers sequentially; this can result in partial scheduling where half of a distributed run occupies GPUs while waiting indefinitely for remaining pods.
• Production Kubernetes deployments running multi-node GRPO therefore add a queueing and gang-scheduling layer such as Kueue or Volcano. Kueue's own documentation introduces waitForPodsReady as "a simple implementation of the all-or-nothing scheduling": "Kueue monitors the workload until all of its Pods are ready" and "if not all pods of the workload are ready within the configured timeout, then the workload is evicted and requeued", while its blockAdmission parameter admits workloads sequentially "to prevent deadlock situations".

Bootstrapping Ray: Slurm sbatch vs KubeRay

Because distributed RL libraries like verl assume an operational Ray cluster already exists before the training script initializes, the mechanical complexity shifts to how Ray is bootstrapped across nodes.

Bootstrapping Ray on Slurm via sbatch

On Slurm clusters, launching Ray requires a batch submission script that handles node rank discovery and explicit networking bindings. Because Slurm executes tasks concurrently across the allocated nodes, the script must parse the SLURM_JOB_NODELIST, identify the lead node rank, start the Ray head daemon with ray start --head, extract the internal GCS port (default 6379), and instruct remaining worker nodes to join via ray start --address. Ray documentation explicitly notes that this setup can be unintuitive because Slurm treats nodes symmetrically, requiring manual shell branching logic.

Furthermore, on multi-tenant HPC systems, Ray's own Slurm guide warns that if two users "both schedule a SLURM job using Ray at the same time, they are both creating a head node", and "as soon as the first head node is created, it will bind some ports and prevent them to be used by another head node", so "users have to manually specify non overlapping ranges of ports" (including --min-worker-port and --max-worker-port). The same guide adds that "on some cluster architecture the network interfaces do not allow to use external IPs between nodes", leaving only internal interfaces such as eth0.

Bootstrapping Ray on Kubernetes with KubeRay

On Kubernetes, the bootstrapping work moves into an operator. Ray's own documentation describes KubeRay as an open-source Kubernetes operator that simplifies the deployment and management of Ray applications on Kubernetes, and it exposes that behaviour through custom resource definitions (CRDs): RayCluster, whose lifecycle (creation and deletion, autoscaling, fault tolerance) KubeRay fully manages; RayJob, where KubeRay automatically creates a RayCluster and submits a job once the cluster is ready, optionally deleting the cluster when the job finishes; RayService, made up of a RayCluster plus Ray Serve deployment graphs with zero-downtime upgrades; and RayCronJob for RayJobs on a recurring schedule. For multi-node GRPO runs, RayJob is the primitive that matches a batch training job.

• RayCluster: Manages head and worker pod pools with automatic health checking and cluster-level lifecycle hooks.
• RayJob: Wraps a RayCluster specification with a defined entrypoint script, automatically managing ephemeral cluster teardown after the GRPO run terminates.
• RayService: Dedicated to zero-downtime inference deployment graphs with high availability routing.
• Operational Trade-off: While KubeRay eliminates custom bash rank-parsing scripts, the platform team must maintain CRD version upgrades, RBAC permissions, and custom container images.

The Interconnect Trap: Why Multi-Node Runs Collapse

A recurring failure mode in multi-node GRPO deployments is a severe throughput collapse when moving from a single 8-GPU node to a multi-node cluster. While single-node execution relies on high-speed intra-node NVLink fabrics (up to 900 GB/s bidirectional per GPU on H100 architectures), multi-node scaling depends entirely on inter-node fabric performance.

In GRPO, the interconnect does not merely handle backward-pass gradient synchronization. Between training iterations, the rollout workers must transmit millions of generated tokens, log-probabilities, and advantage values to the policy trainer, while the policy trainer must broadcast tens of gigabytes of updated model weights back to all rollout workers prior to the next rollouts. This makes communication frequency significantly higher than in pure supervised pre-training.

If the communication library falls back to TCP over standard Ethernet because Remote Direct Memory Access (RDMA) was not initialized, communication latency spikes and GPU compute engines stall waiting for weight synchronization. This failure is silent: the job runs, the loss curve looks plausible, and throughput simply sits far below the single-node baseline because NCCL never bound to the InfiniBand Host Channel Adapters (HCAs).

Proving Fabric Health: All-Reduce Benchmarks

Before launching a multi-node GRPO training run designed to execute for days, infrastructure engineers must empirically validate the health and bandwidth of the distributed fabric. Running nccl-tests (specifically all_reduce_perf) across all allocated ranks produces a bus bandwidth figure that, as NVIDIA's documentation puts it, "should reflect the speed of the hardware bottleneck: NVLink, PCI, QPI, or network", which is what tells you whether the run is actually on the high-speed fabric.

When interpreting benchmark output, engineers must distinguish between Algorithm Bandwidth (algbw) and Bus Bandwidth (busbw). Algorithm bandwidth uses the familiar formula of size divided by time (algbw = S / t). Bus bandwidth applies a per-collective correction factor to the algorithm bandwidth so that, in NVIDIA's words, "we can compare it with the hardware peak bandwidth, independently of the number of ranks used". For an all-reduce operation across n ranks, the formula is:

busbw = algbw * (2 * (n - 1) / n)

For other collective communication primitives used during parameter sharding, the correction factor varies. NVIDIA's summary table gives ReduceScatter, AllGather and AlltoAll a factor of (n - 1) / n, whereas Broadcast and Reduce use a factor of 1. Because the factors differ per collective and depend on the rank count, comparing an uncorrected algbw number against theoretical network limits yields inaccurate conclusions, and an unlabelled bandwidth figure tells you nothing.

• Parameter Sharding Misconception: Teams often assume adopting ZeRO-3 or PyTorch Fully Sharded Data Parallel (FSDP FULL_SHARD) reduces fabric load. In reality, while standard Data Parallelism and ZeRO-1/2 transfer 2*Psi bytes of gradient data per step, ZeRO-3 requires 3*Psi bytes due to re-gathering weights in the forward pass and scattering gradients in the backward pass.
• Fabric Trade-off: Sharding strategies trade interconnect bandwidth for reduced VRAM consumption, intensifying demands on your high-speed network. Review our technical guide on multi-GPU distributed training for interconnect bandwidth calculations.
• Pre-flight Check: Validate with NCCL_DEBUG=INFO and ensure busbw meets expected line rates across all InfiniBand interfaces before launching the Ray runtime.

Handling Node Failures in Long RL Runs

Multi-node GRPO runs frequently extend over multiple days, exposing long-running jobs to transient hardware faults, PCIe dropouts, and thermal throttling. The operational resilience of your stack depends heavily on how the host orchestrator and the in-job Ray runtime handle node loss.

Slurm approaches node failure through an allocation-level model. If a compute node experiences a kernel panic or hardware disconnection, Slurm marks the entire job allocation as failed. Using the --requeue flag in the sbatch definition, Slurm can automatically return the job script to the partition queue, allocate a fresh set of healthy nodes, re-execute the Ray bootstrap sequence, and resume model weights and optimizer states from the most recent distributed checkpoint.

Kubernetes operates on a pod-level reconciliation model. If a node fails, the KubeRay operator detects the missing worker pod and attempts to re-provision it on another available node. However, because distributed PyTorch and NCCL communicators cannot dynamically re-bind communication rings without a global initialization phase, in-flight GRPO training steps cannot simply continue with a newly joined replacement pod. The entire distributed process group must restart.

• State Synchronization: Checkpoints must be committed periodically to high-throughput, shared POSIX or S3-compatible storage.
• Re-initialization Cost: Restarting a 64-GPU RayJob on Kubernetes requires gang scheduling admission via Kueue to ensure the replacement pod is placed before the entire group is initialized.
• Fault Isolation: Slurm simplifies failure recovery on bare-metal by terminating dead node allocations immediately, whereas Kubernetes requires tuned liveness probes and pod disruption budgets to prevent zombie worker states.

Large-Scale GPU Cluster for Multi-Node RL

Executing multi-node GRPO workloads efficiently requires dedicated, bare-metal GPU infrastructure backed by non-blocking high-speed interconnects. For teams scaling reinforcement learning post-training across distributed nodes, that is what the Large-Scale GPU Cluster is built for.

Large-Scale GPU Cluster delivers multi-node allocations, from a handful of nodes up to very large reservations, interconnected with 400 Gb/s InfiniBand NDR fabric, deployed in European data centres in Paris and Finland under strict EU and EEA data sovereignty standards. Available hardware configurations include the NVIDIA H100 (80 GB VRAM), NVIDIA H200 (141 GB VRAM), NVIDIA B200 (192 GB VRAM), NVIDIA B300, NVIDIA GB200, and NVIDIA GB300. Infrastructure provisioning executes in 28 seconds, with flexible contract terms available for 3, 6, 12, or 24 months, as well as custom reservation schedules. Every cluster request receives a quote response within 24 hours, with operational availability covered by a formal SLA agreed individually per business contract.

To eliminate infrastructure management overhead, Lyceum operates and manages either Slurm or Kubernetes directly on your cluster allocation. Your machine learning team selects the orchestration layer that matches your operational pipeline, while the underlying drivers, fabric topology and node health are managed for you, leaving your engineers free to focus on configuring Ray and optimizing GRPO convergence.

For workloads with different operational profiles, Lyceum offers complementary compute options across our European data centres:

• On-demand GPU VM: Provides direct root SSH access to single-node instances with 1 to 8 GPUs linked via intra-chassis NVLink, featuring 18-second provisioning, per-second billing, zero base fees, and no data egress charges.
• Serverless Training: Enables teams to submit containerized fine-tuning and training jobs with starts in under 60 seconds, utilizing direct image pulls from Docker Hub, ECR, or GAR alongside S3-compatible storage integration.