Sam 2 deep dive: why its FIFO memory eviction bothers me (chizkidd.github.io)

Meta has introduced "SAM 2," an advanced model designed for promptable visual segmentation in both images and videos, aiming to enhance real-time video processing. This model leverages a simple transformer architecture combined with a streaming memory system that maintains object identity across video frames by using a FIFO memory bank. This significant upgrade facilitates the segmentation of dynamic scenes, which is crucial for applications in robotics, AR/VR, and autonomous vehicles. SAM 2 is trained on a diverse dataset generated through its novel data engine, the Segment Anything Video (SA-V) dataset, which includes over 1 billion masks collected through an interactive in-the-loop model with human annotators. The introduction of SAM 2 is particularly noteworthy for the AI/ML community as it addresses the limitations of existing video segmentation models, which have struggled to provide robust capabilities for “segmenting anything in videos.” By utilizing a memory attention mechanism, SAM 2 allows the segmentation model to consider both current prompts and past frame features, avoiding the constraints of traditional RNNs. However, the FIFO eviction policy of the memory bank raises concerns regarding the potential loss of critical historical data that could lead to reduced segmentation accuracy when an object reappears after significant changes. Overall, SAM 2 symbolizes a substantial step forward in visual perception technologies, offering a foundation for further research and practical applications in complex video environments.