Charlie Zhou
Senior Research Scientist at Together AI; Founder and Principal Researcher of FutureMLS
Zhou Zhongzhu holds a Ph.D. in Computer Science from the University of Sydney. He is currently a Senior Research Scientist on the Turbo & Kernel team at Together AI and the Founder and Principal Researcher of FutureMLS (Future Machine Learning & Systems Lab). His research focuses on LLM inference acceleration, low-bit quantization, efficient attention architectures, GPU kernels, and the co-optimization of training and inference systems. He has led research projects including OSCAR (2-bit KV Cache Quantization) and CARE (Covariance-Aware MLA Conversion), and contributed to research on efficient training and inference systems such as AURORA. His work has been published at or accepted by leading international conferences, including NeurIPS 2026, ICLR 2026, and ICML 2026. He previously conducted research and engineering in machine learning systems at Microsoft DeepSpeed, Dolby, and Tencent, with a focus on translating algorithmic innovations into reproducible and deployable capabilities for large-scale AI systems.
Topic
Beyond the Paper: Engineering 2-Bit KV Cache for Real-World Deployment
Compressing KV Cache to 2 bits does not automatically translate into equivalent memory savings or performance gains in real-world systems. How can accuracy be preserved over long contexts? How can mixed-precision data be stored in a Paged KV Cache? Can the overhead of quantization and dequantization operators offset the bandwidth savings? This talk centers on OSCAR, a research project led by the speaker, and explores how offline rotation and mixed-precision design reduce accuracy loss under extremely low-bit quantization. From an inference systems perspective, it examines cache layouts, kernel implementation, serving framework integration, and performance validation. Using metrics such as long-context task accuracy, GPU memory consumption, throughput, time to first token (TTFT), and time per output token (TPOT), the talk shares the engineering trade-offs and practical lessons involved in turning a research prototype into a deployable system. Talk Outline Beyond the Paper: Understanding the Real Bottlenecks — Explore KV Cache memory and bandwidth bottlenecks in long-context inference, and examine the gap between theoretical compression ratios and end-to-end performance gains. OSCAR Algorithm Design — Understand how offline rotation, 2-bit quantization, and mixed-precision KV Cache work together to balance compression efficiency and model quality. Accuracy and Reliability — Examine long-context task performance, quantization error across different models, boundary cases, and reproducible evaluation methodologies. Bringing Quantization into Inference Systems — Explore Paged KV Cache layouts, low-bit kernels, quantization and dequantization overhead, and the challenges of integrating with serving frameworks. Evaluating Real-World Deployability — Understand end-to-end trade-offs among GPU memory usage, TTFT, TPOT, throughput, and accuracy, with practical recommendations for deployment. Key Takeaways Understand the key gaps and technical bottlenecks between theoretical compression ratios and actual inference performance gains with 2-bit KV Cache. Learn about OSCAR's offline rotation and mixed-precision strategies, along with methods for evaluating accuracy in long-context scenarios. Understand the cache layout, kernel implementation, and scheduling considerations involved in integrating low-bit KV Cache into Paged KV Cache and inference frameworks. Learn how to assess the practical deployment value of a solution by jointly evaluating accuracy, GPU memory consumption, TTFT, TPOT, and throughput.
