ReplaySSM addresses state-space model bottlenecks by caching inputs instead of state, delivering up to 1.8× faster decoding and 2.1× speculative serving throughput on production models.

NVIDIA and the Dao AI Lab at Princeton University released ReplaySSM, a technique that accelerates decoding in hybrid language models by caching recent inputs rather than storing the full state-space model (SSM) state at every step, according to a paper published October 12, 2026 on arXiv and a blog post by Michelle Horton on the NVIDIA Developer site. The method is already integrated into NVIDIA TensorRT-LLM, vLLM, SGLang and FlashInfer, and runs in production in GLM-5.3-Flash and Kimi-K3 models.

Hybrid models—including NVIDIA Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3 and GLM-5.3-Flash—interleave SSM layers such as Mamba-2, Gated DeltaNet or Kimi Delta Attention with a few Transformer attention layers. SSM layers use a fixed-size state instead of a growing key-value cache, but at serving batch sizes the SSM layers can become the latency bottleneck up to 100,000 tokens of context, according to the October 12 report.

On Nemotron-3-Super-NVFP4 at batch size 256 on one NVIDIA Blackwell Ultra B300 GPU, SSM layers accounted for the largest component of decode-step latency up to 100,000 tokens. Two factors drive this: the model has five times as many SSM layers as attention layers, and one layer's FP32 state is 4 MiB per sequence, equivalent to an FP8 key-value cache for 8,000 tokens per attention layer, the report stated.

Conventional SSM decoding loads the full state from high-bandwidth memory, applies a data-dependent decay and rank-1 update, computes the output, and stores the updated state.

Source link