zHBM: Stacking DRAM directly on the processor
The quest for faster Artificial Intelligence hinges on memory. As AI models grow exponentially, the bottleneck is no longer just processing power, but the sheer volume of data that needs to move at lightning speed. To solve this monumental challenge, industry leaders are embarking on a radical physical redesign of how memory and compute interact, with Samsung laying out an ambitious three-phase roadmap to revolutionize High-Bandwidth Memory (HBM) into a fully integrated memory-and-compute system.
This evolution centers on moving beyond the current 2.5D architecture, which relies on complex silicon interposers to connect processors and memory stacks. Samsung’s vision is to eliminate this physical gap entirely, positioning the processor directly beneath the DRAM. This groundbreaking approach, culminating in their zHBM architecture, promises to unlock unprecedented bandwidth while dramatically slashing power consumption.
The shift begins with a foundational decision: making the base die, the core interface between memory and computation, the primary engine for this transformation. By fabricating the HBM base die on an advanced logic process, Samsung was able to significantly reduce power draw and increase silicon capability, setting the stage for the entire evolution.
Currently, conventional HBM relies on stacking DRAM dies and connecting them via thousands of Through-Silicon Vias (TSVs). While this setup delivers impressive bandwidth, scaling it further faces serious hurdles. As bandwidth demands surge, the physical limitations of TSV signaling speed, area consumption, and massive power requirements become critical constraints. Each new generation of HBM must constantly balance increasing data rates against the mounting thermal and power density challenges.
Samsung’s plan systematically addresses these constraints through three distinct phases. Phase 1 focuses on reclaiming valuable space. By moving memory controller functions onto the base die and redesigning the interface to use smaller die-to-die links, the company frees up silicon area previously occupied by the processor, creating space for expanded compute capabilities. To manage the resulting heat, they introduced specialized Heat Path Blocks to efficiently dissipate energy, demonstrating a focus on practical thermal solutions alongside performance.
Phase 2 moves the focus inward, turning the base die into a smarter subsystem. Here, Samsung plans to populate the unused space with integrated features, such as advanced telemetry and reliability sensors, allowing the memory to communicate its status directly. This phase also explores using the base die for direct memory expansion, potentially offering higher bandwidth solutions than traditional PCIe connections, and integrates select processing elements directly under the DRAM, a concept known as aHBM.
The ultimate destination is Phase 3: zHBM. This is the most radical step, where the processor sits directly beneath the memory stack in a true three-dimensional arrangement. By eliminating the need for the interposer and spatial separation, data transmission is shortened, and the interface circuitry is distributed across the die. This architecture is projected to deliver staggering efficiency gains: estimates suggest zHBM could cut I/O power by around 70% compared to HBM5 and boost DRAM bandwidth by up to 2.3 times while reducing memory power by about 100 watts.
While this path offers a monumental leap forward for AI infrastructure, the engineering challenge remains immense. Achieving zHBM requires mastering advanced wafer-on-wafer bonding and hybrid copper bonding, demanding an unprecedented level of co-design between the DRAM and semiconductor teams to balance the goals of maximum bandwidth, efficiency, and thermal management.
Credit
