Hot Chips 2026: Applying High Bandwidth Flash (HBF) (chipsandcheese.com)

At Hot Chips 2026, discussions centered on High Bandwidth Flash (HBF), a new memory technology that merges flash memory capabilities with the access speed of High Bandwidth Memory (HBM). HBF aims to provide much higher capacity than traditional HBM while still maintaining reasonable bandwidth, positioning it as a potentially transformative solution for machine learning workloads. However, HBF remains conceptual, with no existing products, focusing on simulations and projections for future applications. The notable aspect of HBF is its operational mechanism, requiring software to adapt significantly, akin to managing a low-level disk access API rather than standard memory management practices. HBF's architecture could alleviate the memory capacity constraints faced in modern AI models, particularly for lower-bandwidth workloads. It introduces challenges, such as requiring data retrieval in large, aligned chunks and managing SSD controller-like functions within software, complicating its implementation. Potential use cases discussed included storing model experts and KV caches in HBF to enhance efficiency and reduce cross-device communication issues, which can hinder performance when models are sharded across GPUs. While HBF offers promising capacity advantages, its success will depend on overcoming the substantial software integration hurdles and whether the trade-offs between capacity and bandwidth align with practical AI workloads.