Lila Sciences and Anyscale scaled GLM-5.3-Flash training to 1M tokens per sequence using SkyRL on 64 B200 GPUs, advancing long-context AI capabilities.
Lila Sciences and Anyscale have successfully trained the GLM-5.3-Flash model on sequences of up to 1 million tokens, a significant milestone in long-context AI training. The collaboration utilized SkyRL, an open-source training stack, running on 64 NVIDIA B200 GPUs. This achievement represents a 64-fold increase in sequence length compared to the initial training capacity and addresses the growing need for AI models capable of handling extensive scientific datasets.
The GLM-5.3-Flash model incorporates a hybrid architecture with 34 layers of Kimi Delta Attention (linear attention) and 11 layers of DeepSeek Sparse Attention (DSA). To achieve the 1M-token context, the team implemented several critical optimizations:
These optimizations enabled a peak memory footprint of 144.5 GiB per GPU and a training throughput of 13,700 tokens per second at 1M tokens, with a peak throughput of 24,900 tokens per second achieved at 256k tokens.
The model was fine-tuned on the human genome, using chromosomes 1 and 2 for training and chromosome 22 for evaluation. Post-training at 512k tokens per sequence, the held-out loss improved by 4.8% in 80 steps, with the largest gains observed on tokens with extensive preceding context. This demonstrates the model's ability to leverage long-range dependencies, a critical capability for analyzing structured scientific data such as DNA sequences.
07 across context lengths from 128k to 1M tokens. These results confirm the correctness of the training process while highlighting the practical challenges of scaling long-context models.
Source link
