Together AI launches Together Compiler, achieving up to 3x throughput on AI inference workloads with persistent engine technology.

Together AI has introduced the Together Compiler (TKCC), a system designed to optimize AI inference workloads dynamically and continuously. Announced on November 4, 2026, the compiler leverages high-throughput agents and persistent engine technology to adapt to diverse models, hardware platforms, and traffic patterns, achieving significant performance gains across various benchmarks.

AI inference workloads are increasingly complex, spanning thousands of models, hardware platforms, and traffic shapes. Together AI reports that its infrastructure serves trillions of tokens daily across thousands of customers, each with unique requirements such as batch size, sequence length, and attention sparsity. Traditional manual optimization methods have proven unsustainable in this environment.

TKCC addresses these challenges by employing agents to rewrite GPU code and optimize inference engines. Unlike traditional systems that require lengthy engine restarts for each experiment, TKCC uses persistent engines capable of hot-reloading candidate optimizations. This approach minimizes GPU idle time and increases experimentation throughput by 252 times compared to baseline agentic systems.

In less than 24 hours, TKCC achieved throughput improvements of 1.3 to 3 times across several open-source models, including Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash. Notably, it set state-of-the-art records on SemiAnalysis’s InferenceX benchmark for low-latency performance, reaching 1,111 tokens per second (TPS) on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash. Additionally, TKCC outperformed open-source megakernels by 1.7 times in the Llama 1B batch size 1 setting.

The system also generalizes beyond language model inference, demonstrating applicability in domains such as weather modeling, partial differential equations (PDE), time-series analysis, and biology.

Source link