vLLM is an open-source inference and serving engine for large language models (LLMs). LLM inference has two distinct phases: prefill, which processes the input prompt and builds the key-value (KV) cache, and decode, which uses that cache to generate output one token at a time. These phases have different compute, memory, and scaling characteristics. Prefill-decode (P-D) disaggregation runs them on separate engine instances, allowing each phase to scale and be optimized independently. This can improve resource utilization, throughput, and latency under load. However, it also requires transferring the KV cache from the prefill instance to the decode instance before token generation can begin, making efficient KV transfer critical to time-to-first-token (TTFT).

One of the ways vLLM performs this transfer is through the NVIDIA Inference Xfer Library (NIXL) connector , which moves KV blocks over interconnects that support Remote Direct Memory Access (RDMA), such as AWS Elastic Fabric Adapter ( AWS EFA , the low-latency RDMA network used across servers in AWS). On supported instance types , AWS EFA supports GPUDirect RDMA and Neuron Direct RDMA. NIXL can therefore transfer KV blocks directly between GPU or AWS Trainium device memory without staging through host memory. The default mode for NIXL connector is pull-based : After the prefill instance finishes computing the prompt KV cache, the router relays the source metadata, including the prefill block IDs, to the decode instance. The decode instance then allocates destination blocks, prepares the NIXL transfer, and initiates NIXL READ to pull the KV blocks from the prefill instance.

Source link