NVIDIA's Reshard collective reduces reinforcement learning weight refit latency, addressing inefficiencies in GPU training-inference synchronization.

NVIDIA has unveiled the Reshard collective, a novel communication method designed to optimize weight refit in reinforcement learning (RL) workflows. Weight refit involves transferring updated policy model parameters from training GPUs to inference GPUs, a process critical for generating fresh training trajectories. The Reshard collective aims to address inefficiencies in current methods, particularly as model sizes continue to scale.

Traditional weight refit methods rely on a gather-and-broadcast approach, where training GPUs reconstruct complete model parameters before broadcasting them to inference GPUs. While straightforward, this process results in redundant communication, as inference GPUs receive entire tensors even when only specific shards are required. For large models such as Kimi K3 (2.8 trillion parameters) and Qwen 3.8 2.4T A95B (2.4 trillion parameters), this inefficiency is magnified. Assuming BF16 weights, these models require approximately 11.2 TB and 9.6 TB of communication payload per refit, respectively.

The Reshard collective reformulates weight refit as a DTensor-to-DTensor transfer problem, leveraging PyTorch's distributed tensor abstraction. This approach enables direct shard-to-shard transfers between training and inference GPUs, eliminating the need for intermediate full-tensor representations. By mapping overlapping shard regions and optimizing communication paths, the Reshard collective minimizes payload size and balances data distribution across GPUs.

According to NVIDIA's internal benchmarks, the Reshard collective significantly reduces refit latency. 3 seconds on DGX GB200 systems. 7%. 8 milliseconds compared to flat direct sends.

Source link