Topic

Mooncake + SGLang + EFA: Practical Integration for Large-Scale Prefill-Decode Disaggregated Inference on AWS

With prefill-decode (PD) disaggregation, KVCache transfer across machines has become a critical part of the inference performance path. Mooncake’s layered architecture—where frameworks interact only with a unified Transfer Engine / Store interface, network adapters are pluggable transport plugins, and accelerators are treated as memory locations—turns hardware and framework integration into localized engineering problems. This talk uses Amazon Elastic Fabric Adapter (EFA) as a case study. EFA provides connectionless Scalable Reliable Datagram (SRD) transport but does not support RDMA Reliable Connection (RC). To accommodate these semantics, we redesigned the transport plugin, increasing the peer limit from 48 to over 816 and achieving 97.5% of line-rate bandwidth between GPUs. SGLang, vLLM, and DeepInfer can integrate with Mooncake through a single configuration parameter or fewer than 100 lines of code. The talk will also cover direct EFA access to Trainium HBM and optimizations that reduced Mooncake Store initialization time from 22 minutes to just 40 seconds. The solution is already running in production across a cluster of more than 200 p6-b300 instances, providing practical insights into building high-performance, scalable disaggregated inference systems on Amazon Web Services (AWS). Outline: 1. The Mooncake Project, Community, and Core Design Principles: How layered abstractions turn hardware and framework integration into localized engineering tasks. 2. Why EFA Integration Matters: Understanding PD disaggregation and the differences between SRD and RDMA RC semantics. 3. EFA Transport Plugin Design: Shared-endpoint peer addressing, per-NIC memory registration accounting, NIC affinity, and three silent failure modes. 4. Integration with Inference Frameworks: Integration approaches and deployment lessons from SGLang, vLLM, and DeepInfer. 5. Performance Evaluation: Single-request time to first token (TTFT), phase-by-phase latency breakdowns, concurrency, and long-context workloads. 6. Trainium Integration: Direct EFA access to Neuron HBM and PCIe affinity optimization. 7. Mooncake Store and General Optimizations: Transparent Huge Pages (THP) interleaving, page table entry (PTE) budgeting, io_uring, and unified wheel packaging. 8. Production Experience and Upstream Progress: Lessons from production deployment and the latest upstream development. Audience Takeaways: 1. Understand Mooncake’s layered architecture and learn how to leverage it to integrate new network adapters, accelerators, and inference frameworks. 2. Gain practical configuration guidance, troubleshooting techniques, and solutions to common errors when deploying PD-disaggregated inference with SGLang or vLLM and Mooncake on AWS. 3. Learn why setup overhead becomes the dominant cost once bandwidth is saturated, along with optimization techniques for large memory pool initialization, memory registration, and SSD I/O.