Hot Chips 2026: Flash merges HBM and NAND for massive bandwidth
The quest for cheaper, faster memory in the AI accelerator race has led to exciting conceptual breakthroughs, but the promise of High Bandwidth Flash (HBF) as a replacement for High Bandwidth Memory (HBM) is proving to be more nuanced than initially suggested. While HBF offers staggering capacity at a fraction of the cost, experts suggest it is not a universal panacea; its true value lies in highly specialized, niche applications rather than general performance gains.
When SanDisk introduced the concept of HBF in 2025, the goal was clear: equip AI systems with terabytes of inexpensive memory to reduce dependence on traditional HBM. This vision centered on leveraging 3D NAND technology to offer massive storage density. The system is defined by three performance grades, each balancing capacity and speed, ranging from the basic Grade 1, which delivers 384 GB/s bandwidth, up to Grade 3, which pushes speeds to 3.072 TB/s.
The fundamental appeal of HBF lies in its capacity advantage. It can offer eight to sixteen times the memory capacity of HBM at a comparable cost. However, as demonstrated by modeling large-scale AI workloads, the real performance bottleneck shifts from storage capacity to data throughput—or bandwidth.
In a simulation examining a 72-GPU rack running massive models, the trade-off becomes stark. An HBM-only configuration provided robust aggregate bandwidth, crucial for high-throughput inference. But replacing HBM with HBF dramatically increased the storage capacity, yet simultaneously reduced the aggregate bandwidth to just 922 TB/s. This reveals a critical insight: adding vast memory capacity does not automatically translate into faster processing.
The difference between HBM and HBF bandwidth highlights this conflict. While HBF excels when memory capacity is the limiting factor, HBM remains superior when bandwidth and throughput are the limiting factors for demanding AI computations.
This capacity-bandwidth split points toward a hybrid approach, where HBF handles the bulk storage while HBM manages the critical, high-speed data transfer. For instance, in Mixture-of-Experts (MoE) models, HBF could be perfectly suited to store massive pools of less frequently accessed expert weights, which are essentially write-once and relatively cold. This strategy allows the frequently accessed, active weights to remain in the faster HBM cache, optimizing the overall system.
Furthermore, HBF offers intriguing solutions for long-context inference. Sparse-attention models use only small portions of their large context caches during each decoding step. HBF could effectively store this large Key-Value (KV) cache, allowing the accelerator to fetch only the required data from HBF to HBM as needed.
However, realizing these benefits requires more than just hardware innovation; it demands a complete software overhaul. The complexity arises because achieving maximum bandwidth requires massive data transfers over DMA, and the software stack—including frameworks like vLLM—must intelligently manage which data resides in HBM versus HBF, handle data placement, and monitor the write endurance of the flash memory.
For HBF to succeed beyond a niche application, hardware vendors like AMD and Nvidia must support it, and inference developers must implement the necessary memory allocation and data management policies. Ultimately, while HBF is a nascent technology, it is not a replacement for HBM. The consensus is that HBF‘s role is not to sit directly next to the accelerator, but rather to function as a secondary memory tier—a specialized solution for the bulk storage needs of the AI ecosystem.
Credit
