Benchmarking null-space projection for continual learning in SmolLM-135M (zenodo.org)

The recent release of the "Wake-Sleep Null-Space Consolidation (WS-NSC)" framework marks a significant advancement in continual learning for Large Language Models (LLMs), particularly in mitigating catastrophic forgetting during sequential adaptation. Traditional methods often face challenges with memory footprint and representation corruption when integrating low-rank updates. WS-NSC addresses this issue by employing an algebraic consolidation technique that projects updates onto the null-space of historical activation covariances. This empirical study, applied to the SmolLM-135M model across multiple datasets, showcases WS-NSC's ability to significantly reduce catastrophic forgetting (just -0.39% relative drift) compared to direct merging methods (which experienced a +5.83% drift). However, the approach introduces a trade-off between stability and plasticity. While it effectively halts anchor drift, it also constrains the model's ability to retain task gains, with performance drops noted at 37.7% and 24.4% based on retention metrics. These findings formalize the complexities of managing parameter update norms and functional retention, offering crucial insights into the ongoing development of more resilient and efficient continual learning systems in AI. The accompanying code is available for researchers looking to implement and build upon these findings.