ReplaySSM, a joint project by NVIDIA and Princeton's Dao AI Lab, reduces state memory traffic in hybrid language models by caching recent inputs rather than full state matrices, delivering up to 1.80× kernel speedups at serving batch sizes.

NVIDIA and Princeton University's Dao AI Lab published ReplaySSM on October 12, 2026, a technique that addresses memory-bandwidth bottlenecks in hybrid language models by caching recent inputs instead of storing full recurrent state at every decoding step. The authors report kernel speedups of 1.46× to 1.80× and whole-decode-step speedups of 1.20× to 1.47× across four models at batch size 256, according to the paper ReplaySSM: Cache SSM Inputs, Not State available on arXiv and the NVIDIA Developer Blog post by Michelle Horton.

Hybrid models interleave state-space model (SSM) layers—such as Mamba-2, Gated DeltaNet, or Kimi Delta Attention—with a smaller number of attention layers. NVIDIA Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3, and GLM-5.3-Flash all follow this architecture. SSM layers use fixed-size recurrent state rather than key-value caches that grow with context length, a design intended to accelerate decoding.

At serving batch sizes, however, SSM layers can become the latency bottleneck. On NVIDIA Nemotron-3-Super-NVFP4 at batch size 256 on one NVIDIA Blackwell Ultra B300 GPU, SSM layers accounted for the largest component of decode-step latency up to 100,000 tokens of context, according to the October 12 blog post. Each conventional SSM decoding step loads the full state from high-bandwidth memory, applies a data-dependent decay and rank-one update, computes the output, and writes the updated state back.

Source link