d-Matrix achieves 20x bandwidth density over NVIDIA Rubin in new chip

Noah Miller 08/25/2026

When we talk about pushing the boundaries of artificial intelligence, we often focus on raw processing power. But behind the scenes, the real challenge isn’t just about how fast a chip can calculate; it’s about managing the flow of information. This distinction is crucial when analyzing the workload during AI inference—the process where a trained model actually generates an output.

The entire inference process can be neatly divided into two distinct phases: the prefill phase and the decode phase. While both phases require significant computational effort, they face fundamentally different bottlenecks, demanding specialized strategies from hardware designers and software engineers.

The prefill phase, which deals with feeding the model the initial input prompts, is often dominated by the sheer amount of computation. In this stage, having access to massive amounts of processing power is highly advantageous. You can deploy all the compute in the world, and for this initial stage, that is exactly what you need to maximize speed.

However, the true bottleneck emerges in the subsequent decode phase. This is where the model starts generating the actual output, and here, the story changes dramatically. Decode is overwhelmingly the more time-consuming portion of the entire workload because it is strictly memory-bandwidth bound.

This means that having access to unlimited compute power is, in many scenarios, not enough. For the decode phase, the limiting factor shifts from raw processing speed to how quickly data can be shuffled and delivered between the processor and the memory. It’s not about how fast the brain can think, but how fast it can read its notes.

Because of this critical difference in performance requirements, researchers and developers are now exploring various strategies to bridge this gap. They are looking for novel ways to optimize the memory architecture and data flow, ensuring that the system doesn’t stall waiting for data, allowing AI systems to truly unleash the potential of their massive computational resources.