Nvidia reveals Groq 3 LPX architecture and first benchmark at Hot Chips 2026
The AI hardware landscape just got a serious shake-up. At the Hot Chips 2026 conference, a moment of genuine cross-industry innovation was showcased when the former chief architect of Groq presented their proprietary inference chip as Nvidia silicon, signaling a massive integration of disparate AI philosophies.
The focus quickly shifted to the architecture powering Groq’s performance. Igor Arsovski, Nvidia’s VP of hardware, detailed the design of the Groq 3 LPX rack, revealing a system built around a radical memory approach. This architecture is not just fast; it’s fundamentally different. The hardware utilizes SRAM without HBM, embedding critical model weights directly onto the chip. This design choice eliminates the latency issues associated with streaming data from external memory, allowing for a fully deterministic and highly efficient operation.
Benchmarks confirmed the promise of this novel approach. Artificial Analysis measured the hardware performing a demanding 100K-context Gemma 4 31B reasoning workload, achieving 3,431 output tokens per second. This rate roughly quadrupled the performance of the next-fastest public endpoint available, demonstrating exceptional efficiency in high-context reasoning.
The engineering behind this speed is rooted in deterministic execution. By allowing the compiler to schedule operations at the clock-cycle level, the hardware can manage power draw and thermal constraints with surgical precision. This allows the system to cut voltage droop by more than 60% and effectively equalize heat across the chips, leading to an impressive 10 to 11% performance boost under the same thermal limit.
The physical specifications underscore this breakthrough. Each chip carries approximately 500MB of on-die SRAM, providing a staggering 128GB of memory across a full 256-chip LPX rack. This configuration delivers an aggregate bandwidth of 40 PB/s and chip-to-chip latency of just 350 nanoseconds within a Vera Rubin-compatible, liquid-cooled rack environment. This architecture is designed for density and speed, prioritizing the immediate access of weights over the traditional streaming methods.
This design philosophy moves away from the typical GPU paradigm, where memory constraints often dictate the design. The result is a system focused on raw inference throughput, where capacity is achieved through optimized on-die memory rather than reliance on external High Bandwidth Memory. Nvidia’s move, including the acquisition of Groq’s IP, positions this approach at the forefront of the next generation of specialized AI accelerators.
Beyond the chip design, the system is designed for sophisticated deployment. Nvidia is pitching the LPX rack as a powerful decode co-processor that complements the compute power of Rubin GPUs. By strategically splitting the workload—using the LPUs for token generation while the GPUs handle heavy prefill and KV cache building—the system achieves gains three to five times over Rubin alone. This collaboration, bolstered by specialized FPGAs and new runtime extensions, is redefining how massive AI models are deployed and executed in data centers.
Meanwhile, competitors are also pushing the boundaries. Cerebras recently unveiled its CS4 system, which demonstrated its own path to extreme performance, achieving up to 30 times the speed of GPUs and offering tenfold the token capacity. This rivalry is driving the industry toward architectural solutions that prioritize specialized memory and deterministic performance, proving that the future of AI hardware lies in innovative specialization.
Buy on Amazon
Credit
