Moonshot AI's Kimi K3.1 Pro, a 2.8T-parameter model, is deployed on AMD Instinct MI355X GPUs with ATOM and vLLM support for multimodal AI tasks.
Moonshot AI has announced the deployment of its latest multimodal AI model, Kimi K3.1 Pro, on AMD Instinct MI355X GPUs. The model, which features 2.8 trillion total parameters and 104 billion activated parameters, is designed to handle complex AI tasks with a one-million-token context window. This deployment leverages AMD's ATOM inference engine and vLLM for optimized performance and scalability.
Kimi K3.1 Pro retains the hybrid architecture of its predecessor, Kimi K3, with 69 Kimi Delta Attention (KDA) layers and 24 latent-attention layers. It introduces a sparse indexer to each latent-attention layer, enabling the selection of 2,048 tokens of historical context per query. The model operates with an 896-expert mixture-of-experts (MoE) architecture, where 16 experts are activated per token, and uses MXFP4 quantization-aware training for efficiency.
The model supports both text and image inputs, with a MoonViT-V2 vision encoder comprising 401 million parameters. Additionally, Kimi K3.1 Pro includes a DSpark speculative-decoding draft, which accelerates token generation by proposing draft tokens in parallel with the main decoding process.
1 Pro on eight MI355X GPUs, utilizing tensor parallelism for efficient execution. The released MXFP4 checkpoint is loaded without conversion, and the system supports both long-context and low-latency interactive workloads. During startup, memory usage peaks at approximately 199 GB per GPU out of the available 288 GB, allowing a single one-million-token request to fit on one node.
Source link
