Topic

Agent-Era Inference Infrastructure: Full-Stack Intelligent Inspection and Root Cause Diagnosis in Practice

Inference is evolving from single requests to long-running, multi-step workflows driven by AI agents, pushing failures deeper into the infrastructure stack. Routing imbalance at the gateway, KV cache fluctuations in the inference engine, and cold-start latency spikes in the sandbox can all manifest as unexplained P99 latency spikes, leaving threshold-based alerts ineffective. This talk presents Alibaba Cloud’s practices in intelligent inspection and diagnosis for large-scale inference systems. Using UModel, it models the resource layer, inference engine, gateway, and AgentSandbox as a unified graph that supports reasoning. On this foundation, it builds an agent-driven, adaptive health inspection system for large-scale inference infrastructure, along with root cause analysis (RCA) backed by traceable evidence. Outline: 1. Observability Challenges Across the Inference Stack, from Sandbox and Agents to Inference Services 2. End-to-End Observability Data Collection Design 3. Establishing a Unified Global Data Model: UModel 4. Supporting Large-Scale Inspection Through Health Score Calculation 5. Design Principles for Inspection and Diagnosis Agents 6. Demonstration of Practical Results Key Takeaways: 1. A practical checklist and engineering solutions for collecting full-stack inference observability data, with a focus on cross-layer traceability and retaining data for short-lived entities. 2. A unified modeling approach for inference infrastructure using UModel, covering entity relationships, stable IDs, time travel, and change modeling. 3. An actionable health scoring framework featuring peer-based baselines, topology-aware aggregation, and graded identification of degraded health—and how it can reduce the cost of large-scale inspection by an order of magnitude. 4. A complete design for inspection and diagnosis agents, covering architectural layering, context pruning, and trust constraints, as well as the use of fault injection to quantitatively measure RCA accuracy.