Ruiyang Ma
Member of the Xiaohongshu Dots Team, Ph.D. Candidate at the School of Computer Science, Peking University
Ruiyang Ma is a Ph.D. student at the School of Computer Science, Peking University, supervised by Professor Guojie Luo. He was selected for Xiaohongshu’s Dots-Ace elite internship program and is a core team member responsible for building and researching inference infrastructure for Agentic workloads in Dots models. His doctoral research spans LLM inference software stacks, emerging computer architectures and interconnects, and electronic design automation (EDA). He has contributed to the development of several open-source projects, including SGLang HiCache/HiSparse, Mooncake, and Dynamo.
Topic
Xiaohongshu Dots: Building Inference Infrastructure for Agentic Workloads and Real-World Applications
Agentic applications involve multi-turn interactions, tool calls, and long-context processing, introducing new challenges for inference infrastructure in cache reuse, request scheduling, and resource utilization. This talk will share Xiaohongshu Dots’ experience in building inference infrastructure for Agentic workloads. It will explain how Dots leverages Dynamo to establish a global view of resources and caches, combines KV-aware routing with load balancing, and enables session-level cache management through programmable KV control and the Session Prefix Indexer. The talk will also explore how SGLang and Mooncake work together to provide underlying execution, storage, and data transfer capabilities. Drawing on two real-world use cases—online serving for the Diandian Agent and post-training for Agent Reinforcement Learning (Agent RL)—the presentation will discuss architectural trade-offs, practical implementation lessons, and optimization strategies for reducing interactive latency, increasing sampling throughput, and improving overall resource efficiency. Talk Outline Infrastructure Requirements and Challenges in Agentic Workloads Overall Architecture and Core Capabilities of Dots Agentic Infrastructure Routing Strategies Based on Global Views, KV Awareness, and Load Balancing Design of Programmable KV Control and the Session Prefix Indexer Online Serving for the Diandian Agent: Applications and Efficiency Optimization Agent RL Post-Training: Applications and Efficiency Optimization Key Takeaways Understand the core inference infrastructure requirements of Agentic workloads, along with Dots’ architectural design and the collaborative roles of Dynamo, SGLang, and Mooncake. Learn the design principles and engineering trade-offs behind global resource and cache views, KV-aware routing, load balancing, programmable KV control, and session-level prefix indexing. Gain practical insights from online serving for the Diandian Agent and Agent RL post-training, including techniques for reducing interactive latency, increasing sampling throughput, and improving resource utilization.
