Tencent has released a major open-source upgrade to its HPC-Ops AI inference library, delivering dramatic performance improvements that could benefit AI researchers and developers worldwide. The library now incorporates the company’s Stem sparse attention algorithm — recently accepted for presentation at the prestigious International Conference on Machine Learning (ICML 2026) — which in combination with HPC-Ops’ optimized operators reduces first-token latency by up to 3.7x under 128K context windows.
First-token latency — the time an AI system takes to begin generating its first output after receiving a query — is a critical metric for AI-powered applications, particularly those where users interact with AI in real time. A 3.7x improvement means that AI applications built on HPC-Ops can feel dramatically more responsive, transforming user experience in conversational AI, coding assistance, and interactive analysis tools.
The open-source release reflects Tencent’s strategy of contributing to the global AI research community while building goodwill and developer adoption for its AI infrastructure. By sharing these performance optimizations, Tencent enables companies of all sizes — including startups and academic institutions — to benefit from inference optimizations that would otherwise require significant engineering resources to develop independently.
The Stem sparse attention algorithm at the core of this release addresses one of the fundamental computational challenges in large language models: the quadratic scaling of attention computation with sequence length. By using sparse attention patterns that selectively compute attention only where it matters most, Stem dramatically reduces the computational burden of processing long contexts while maintaining the quality of AI outputs.
HPC-Ops itself has evolved significantly, transforming from a single high-performance operator library into a comprehensive system-level inference optimization suite. The upgraded library now covers multiple dimensions of AI inference performance: computational operators, memory management, batching strategies, and hardware utilization — creating a unified platform for achieving production-grade performance across diverse AI workloads.
The ICML 2026 acceptance of the Stem algorithm is particularly significant, as it validates the scientific rigor of Tencent’s approach and positions the company alongside leading AI research institutions. This combination of cutting-edge research and practical open-source engineering tools demonstrates how leading technology companies can advance both the science and engineering of AI simultaneously — accelerating the broader ecosystem’s ability to build faster, more efficient AI applications.
