Face-to-face bonding: Stacking AI chips for 100 TB/s speed
The race to power artificial intelligence is currently running on a tightrope, constantly battling the relentless demand for faster, more efficient processing. At the core of this high-stakes endeavor is the need to eliminate bottlenecks—specifically, the lag between processing speed and memory bandwidth. Enter d-Matrix and its groundbreaking Raptor architecture, a solution that rethinks the very geometry of how compute and memory communicate.
d-Matrix presented Raptor, touted as the first 3D DRAM accelerator for generative inference, at Hot Chips 2026. This technology achieves staggering performance by bonding a TSMC 4nm compute die face-to-face with a custom-designed DRAM die. This novel arrangement delivers a phenomenal 100 terabytes per second of bandwidth, sourced from 32 gigabytes per card.
The magic lies in the vertical interface. By inverting the traditional 3D stacking arrangement, the logic die sits on top, with the DRAM die serving as the interposer that carries critical PCIe and die-to-die signals via TSVs. This structural innovation allows for exponential bandwidth gains, enabling the system to drive data up to 100 terabytes per second for the same amount of power. This efficiency is remarkable: the energy cost for the vertical interface clocked in at just 0.37 pJ/bit, significantly lower than the roughly 2.4 pJ/bit required for moving data into traditional HBM4 base dies.
But achieving this speed comes with some serious engineering hurdles. The custom DRAM die is designed to operate at elevated temperatures (up to 105°C), which normally causes memory retention to collapse. d-Matrix ingeniously countered this by redesigning the memory banks to be smaller and more resilient, successfully absorbing the thermal penalty while managing data integrity. This sophisticated co-design—where the bank geometry, spare-bank muxing, and error correction were tailored specifically to the compute die—is what yields these efficiency numbers, a level of integration unseen in current memory catalogs.
Further demonstrating the complexity is how the system handles data integrity. The interface employs a sophisticated technique that compares data flits and stores inversion tags, recovering a significant portion of the power savings that traditional data bus inversion would offer. While the vertical interface still draws a substantial 296W, it manages this power density effectively, achieving only about 0.5W per square millimeter, a feat manageable by advanced liquid cooling.
Despite these leaps in hardware efficiency, the ultimate bottleneck in large-scale AI inference is shifting from raw bandwidth to communication overhead. As models grow larger, the necessity of synchronizing data across multiple cards begins to consume precious bandwidth. Experts note that as models evolve, the challenge shifts from achieving blazing-fast data transfer to mitigating the increasing overhead of inter-card synchronization, suggesting that while the physical connection is optimized, the logical flow of massive AI workloads remains a complex challenge.
The future of this technology hinges on the supply chain. While d-Matrix has established partnerships with industry giants like TSMC and Alchip, the identity of the DRAM fabricator remains undisclosed. Given the explosive growth in demand for high-end memory—with price estimates suggesting a massive increase between 2024 and 2026—the ability to scale Raptor‘s volume depends on securing this critical supply. The speed of innovation in AI hardware is undeniable, but the path to mass deployment is now inextricably linked to solving these complex manufacturing dependencies.
Credit
