Together AI introduces Together Compiler, achieving up to 2.6x higher throughput for AI inference with persistent engines and agent-driven optimization.
Together AI has announced the launch of Together Compiler (TKCC), a groundbreaking system designed to optimize AI inference workloads through continuous compilation. The system, introduced on November 4, 2026, aims to address the challenges posed by the dynamic nature of AI workloads, which span diverse models, hardware platforms, and traffic patterns.
AI inference workloads are increasingly complex, with Together AI serving trillions of tokens daily across thousands of customers. These workloads vary significantly in batch size, sequence length, sparsity, and other factors. Additionally, the hardware landscape is highly fragmented, with platforms such as NVIDIA H100, B200, and Groq requiring distinct optimization strategies. Together Compiler leverages agent-driven optimization to adapt to these diverse conditions without relying on static rules or manual tuning.
TKCC introduces "persistent engines" that eliminate the need for repeated engine startups during optimization. Traditional systems require up to 467 seconds for setup, while TKCC reduces this to just 9 seconds. This innovation enables a 252x increase in experiment throughput compared to baseline agentic systems. Agents within TKCC share optimization proposals via a message board, ensuring continuous testing and minimizing GPU idle time.
1-Flash. 1-Flash. 7x in Llama 1B batch size 1 settings.
Source link
