32 TB RAM for your GPUs: Marvell shows revolutionary Photonic Fabric Memory Module that uses light to link up to 16 GPUs and slash CAPEX — but Nvidia won't stick around doing nothing (www.techradar.com)

Marvell has unveiled its innovative Photonic Fabric Memory Module, aiming to revolutionize AI infrastructure by enabling a shared memory system across multiple GPUs. This module can connect up to 16 GPUs and provides a total of 32TB of DDR5 and 1,152GB of HBM3e memory via optical interconnections. By utilizing light instead of traditional copper, Marvell's design allows substantial memory to be stored externally from processors, potentially transforming how large language models manage and retain extensive contextual data. The memory appliance, which is still in pre-silicon development, is targeted for customer sampling by 2027. This announcement is significant for the AI/ML community as it addresses the growing demand for scalable memory solutions, particularly for applications requiring large amounts of working memory, like KV caches in AI workloads. Marvell's approach could alleviate the pressure on capital expenditures (CAPEX) by enabling users to purchase memory independently from GPUs, a paradigm shift as HBM becomes increasingly expensive. However, the market landscape is heating up with Nvidia already investing in Marvell and developing its own memory solutions, highlighting the competitive race to solve the challenges of AI memory architecture. As both companies move forward, the implications for performance and cost-efficiency in AI processing will be closely watched.