Topic

Sparse Attention-Driven KVCache Management for Domestic AI Supernodes

In long-context inference, persistent KVCache occupancy in High-Bandwidth Memory (HBM) limits service concurrency and decoding throughput. This talk presents a sparse attention-driven KVCache management system designed for domestic AI supernodes. The system uses Top-K indices to enable on-demand, token-level data access. Complete historical KV data is stored in a host memory pool within the supernode, while hot data is cached on the NPU and the data required for the current computation is assembled on demand. An index-based copy operator enables compatibility with graph execution on NPUs. The talk also shares integration experience with SGLang, including a redesign of the supernode-based KVCache memory pool, and discusses approaches to capacity scaling and performance optimization for long-context inference. Agenda Bottlenecks in Long-Context Inference: Analyze how KVCache capacity limits decoding concurrency, and the mismatch between page-level data movement and token-level sparse access. Sparse-Driven System Design: Introduce the host storage, device cache, and index mapping of SparseKVCacheManager, and explain how the UniDex index-copy operator enables compatibility with NPU graph execution. SGLang Integration and Performance Analysis: Share integration experience on domestic NPUs, analyze the relationship between HBM savings, larger decoding batches, and throughput improvements, and examine the overhead of on-demand data movement. Scaling for Domestic AI Supernodes: Present validation of remote DRAM sparse access based on HCCS and MemFabric, and discuss shared KV memory pool design and memory utilization in scenarios with separated prefill and decoding. Key Takeaways Understand the capacity and bandwidth bottlenecks of KVCache in long-context inference, and learn how sparse access can reduce device-resident data and increase service concurrency. Learn how Top-K index-driven, token-level KV management and data movement work, as well as engineering approaches for adapting dynamic sparse access to NPU graph execution. Understand the architectural considerations behind a shared KV memory pool for domestic AI supernodes, including how high-speed interconnects, remote DRAM access, and cross-node index addressing can work together to scale available memory capacity. Learn from SGLang integration and performance analysis how to evaluate the applicability of offloading solutions based on HBM utilization, data-transfer overhead, decoding throughput, and request latency.