Nvidia optimizes Rubin for GPU to rack inference performance

The AI world is entering a thrilling new phase, and it’s not driven by massive frontier training runs anymore. The focus is shifting sharply toward agentic AI inference—the demanding need to deliver generated tokens quickly, efficiently, and at a low unit cost. This paradigm shift is precisely why Nvidia’s upcoming Vera Rubin platform, arriving later this year, is poised to redefine what’s possible in large-scale data center computing.

While the raw power of Vera Rubin systems is immense, the real story lies in how the architecture manages these complex demands. The new platform is designed not just for brute force, but for intelligent efficiency, pushing performance from the GPU level all the way up to the entire rack and data-center scale.

At the heart of this evolution are significant architectural refinements aimed squarely at maximizing inference efficiency. Nvidia is showcasing how the Rubin architecture tackles the growing complexity of modern models, particularly Mixture-of-Experts (MoE) architectures that are becoming the standard for massive language models.

A key component in achieving this efficiency is the Tensor Memory Accelerator (TMA), which has been significantly upgraded to handle the demands of MoE models. Previously, managing the memory descriptors for numerous experts created significant overhead. The improved TMA now allows for a single, unified MoE descriptor to be managed directly at runtime, drastically reducing computation and freeing up valuable GPU cycles for actual inference calculations.

Further boosting performance are fundamental improvements to matrix operations within the Tensor Cores. By doubling the throughput on the K dimension—the shared inner dimension of a matrix multiplication—Rubin enables faster processing for critical context handling and decode phases in LLMs. This refinement translates directly into greater overall throughput and lower latency.

The attention mechanism, foundational to transformer models, also sees a massive jump in speed. Nvidia has boosted the performance of the Softmax operation within the GPU’s Special Function Unit (SFU), delivering up to a 4X boost compared to previous architectures for these essential calculations.

This attention innovation is particularly impactful for long-context models. By speeding up the core mathematical operations, Rubin keeps pace with the need for context lengths stretching into the millions of tokens without sacrificing speed or efficiency.

Beyond computation, Vera Rubin addresses bottlenecks in communication across large systems. The NVLink fabric connecting GPUs within a rack now operates with vastly reduced overhead. Instead of relying on complex memory barriers and synchronization flags, the new architecture introduces counted writes, which streamline inter-GPU data sharing. This innovative approach cuts down on network traffic and latency, ensuring that model weights and critical information move across the fabric with maximum efficiency.

By combining superior single-threaded performance from the Vera CPUs with these optimized GPU features, Nvidia is creating a system that doesn’t just deliver raw speed. It promises to lower per-token inference costs while simultaneously skyrocketing the performance of agentic workflows, setting a new benchmark for what massive-scale AI can achieve.