The Continual Learning Problem (jessylin.com)

Researchers propose using sparse memory layers as a practical architecture for parametric continual learning and demonstrate strong empirical gains in reducing catastrophic forgetting. In "Continual Learning via Sparse Memory Finetuning" they show finetuning memory layers retains performance far better than standard approaches: when training on TriviaQA facts, downstream NaturalQuestions accuracy falls 89% with full finetuning, 71% with LoRA, but only 11% with memory layers. The work frames continual learning as two coupled problems—generalization (what to learn from each experience) and integration (how to update old knowledge without breaking it)—and argues memory layers address the latter while leaving richer self-supervision/augmentation strategies (e.g., Active Reading) to help the former. Technically, a memory layer replaces an FFN with a learned key/value pool of N slots (N ~ 1M–10M) indexed by a projected query q(x). A top-k lookup (k≈32) retrieves a tiny subset of slots, producing y = softmax(K_I q)V_I; an input-dependent gating (silu and W1/W2 projections) composes the final output. The result is high-capacity but extremely sparse per-step updates (only ~0.03%–0.0002% of memory parameters active), enabling targeted updates, learned routing and more scalable lifelong compression than replay, RAG, or traditional PEFT. This architecture bridges ideas from MoEs and retrieval while allowing the model weights themselves to continually integrate new knowledge without proliferating discrete adapters or unbounded buffers.