Multi-turn agentic workloads build on accumulated context as an agent reasons, invokes tools, and incorporates earlier results. With each new turn, the client resends the conversation history, including system instructions, prior prompts, tool results, and model responses, causing the growing context to be processed repeatedly. In Prefill-decode (P-D) disaggregation, each request is split across specialized nodes: the prefill node processes the prompt and builds its key-value (KV) cache, then the decode node retrieves those KV blocks and generates the response.

Conventional disaggregated serving, however, only transfers KV cache in one direction: from prefill to decode node. This one-directional flow creates two inefficiencies for multi-turn workloads. First, after generating a response, the decode node holds KV cache entries for both the prompt and the newly generated tokens. When processing the next turn, the prefill node cannot retrieve those entries, so it must recompute the KV projections for the previous response tokens. Second, the decode node maintains the session state while the prefill nodes don’t, increasing the probability of cache eviction for active sessions on the prefill nodes. On a full cache miss, the prefill node must recompute KV projections for the entire conversation history, including system instructions, prior user requests, tool results, and model-generated responses. As the conversation grows, this redundant work consumes more accelerator capacity and increases time to first token (TTFT).

When you build conversational assistants and agentic applications, longer interaction histories can increase latency and consume accelerator capacity through repeated prefill computation. In this post, we show how bi-directional KV cache transfer lets prefill and decode nodes reuse previously computed context across turns, which improves accelerator utilization and helps applications sustain responsive TTFT as conversations grow.

Source link