Topic

Large Language Model Inference Scheduling and Cache Optimization for Multi-Agent Applications

Multi-agent applications are characterized by complex execution dependencies, frequent tool calls, and intensive context interactions. Although LLM inference systems support batching and cache reuse, they still face challenges such as resource contention among critical tasks, idle GPU memory during tool-waiting periods, and redundant cache storage across agents. This talk presents two systems designed to address these challenges. TokenCake jointly optimizes task scheduling and cache management by accounting for agent dependencies and runtime states. It combines dynamic GPU memory partitioning, proactive cache offloading, and predictive prefetching to reduce critical-path blocking and context recomputation. TokenDance targets shared contexts in turn-based collaboration. By combining set-based cache reuse, differential storage, and fused reconstruction, it reduces redundant computation and storage overhead across agents. Experiments demonstrate that these approaches can substantially reduce end-to-end latency in multi-agent applications while improving resource utilization and concurrent serving capacity under limited GPU memory. Talk Outline 1. Background: Inference Challenges in Multi-Agent Collaboration Illustrate typical execution workflows—including task decomposition, tool calls, result aggregation, and multi-turn interactions—through collaborative code generation, deep research, and social simulation. Analyze the main factors affecting application performance: task dependencies determine execution order, tool latency creates resource idle windows, and growing contexts increase computational and storage pressure. Establish an application-level performance perspective that extends beyond individual model-call latency to task completion time, per-turn latency, and the concurrency levels achievable while meeting service-level objectives. 2. Bottleneck Analysis: Understanding Resource Inefficiency Through KV Cache Temporal dimension: Caches continue to occupy GPU memory while agents wait for external tools, whereas discarding them increases subsequent recomputation costs. Spatial dimension: Multiple agents compete for GPU memory, and non-critical tasks may consume cache capacity needed by critical-path tasks. Sharing dimension: Multiple agents may receive identical content but have different private histories and content positions, limiting reuse through conventional prefix caching. Analysis methodology: Use execution timelines, dependency graphs, and cache lifecycles to identify the sources of waiting, resource contention, recomputation, and redundant storage. 3. TokenCake: Agent-Aware Scheduling and GPU Memory Management Representing application-level information: Convey agent dependencies, tool-call events, and runtime states to the inference system. Spatial scheduling: Determine task priorities based on dependency graphs and runtime states, using dynamic GPU memory partitioning to ensure critical tasks can progress. Temporal scheduling: Proactively offload idle caches during tool-call windows and use predictive prefetching to reduce data-transfer delays when execution resumes. Co-optimization: Analyze the trade-offs among task priority, GPU memory pressure, and migration costs, explaining how scheduling and cache management jointly affect end-to-end performance. 4. TokenDance: Turn-Level Cache Sharing and Differential Storage Collaboration patterns: Examine the All-Gather interaction workflow in which agents generate results, aggregate them centrally, and redistribute the shared context. Set-based reuse: Preserve the logical block structure of prompts and process compatible requests in groups to amortize the overhead of redundant cache reuse. Differential storage: Reduce duplicated storage of similar caches through shared primary copies and agent-specific deltas. Fused reconstruction: Integrate delta reconstruction into cache transfer to reduce additional data movement overhead. Applicability and limitations: Discuss how the proportion of shared context, request compatibility, and output quality affect optimization benefits. 5. Engineering Practice and Case Studies Integration pathways: Explain the information exchanged among the application orchestration layer, inference scheduler, and cache management module, along with integration strategies for existing inference engines. Case studies: Examine critical-path optimization in code generation and deep research, and multi-turn sharing and concurrency scaling in GenerativeAgents and AgentSociety. Evaluation methodology: Measure end-to-end latency, tail latency, cache occupancy, migration overhead, concurrency capacity, and task quality to avoid overlooking system costs through reliance on a single metric. Engineering trade-offs: Explore how to select strategies, establish fallback mechanisms, and validate performance gains as tool latency fluctuates, cache-sharing patterns change, and workloads grow. 6. Methodological Takeaways: Optimizing Inference Systems Around Application Collaboration Structures Translate dependencies, tool-waiting periods, and context sharing into actionable information for underlying inference systems. Jointly consider task scheduling, cache lifecycles, and data representations to identify opportunities for coordinated optimization. Establish an end-to-end optimization workflow: workload analysis, bottleneck identification, mechanism design, and end-to-end validation. Key Takeaways This talk uses two systems to demonstrate how multi-agent collaboration patterns can be translated into measurable performance gains: TokenCake: Under the GPU-memory-constrained configurations evaluated in the paper, it reduces end-to-end latency by up to 47.06% compared with vLLM. TokenDance: Across the evaluated workloads and under the same serving-latency constraint, it supports up to 2.7× as many concurrent agents as vLLM with prefix caching enabled. Attendees will gain an end-to-end understanding of the optimization process, from bottleneck analysis and mechanism design to experimental validation, and learn how to evaluate the trade-offs involved in improving inference efficiency for multi-agent applications.