New technique makes AI models leaner and faster while they're still learning (news.mit.edu)

Researchers from MIT's CSAIL, along with partners from Max Planck and other institutions, have introduced CompreSSM, a groundbreaking technique that enables faster and leaner training of AI models by compressing them during the learning process rather than after. This method particularly targets state-space models—used in areas from language processing to robotics—allowing models to shed unnecessary components early in training. By employing concepts from control theory, such as Hankel singular values, CompreSSM identifies and discards less important model parts after only 10% of the training cycle, resulting in significant training speed-ups and comparable accuracy to larger models. The significance of CompreSSM lies in its ability to integrate compression into the training phase, contrasting sharply with conventional methods that incur high computational costs by first fully training a large model before reducing its size. With impressive results, such as achieving an approximately 4x training speed and high accuracy retention in benchmarks, CompreSSM demonstrates a promising new approach for large-scale AI model training. As a theoretically grounded method, it opens up opportunities for broader applications in emerging architectures, including those pivotal in today’s AI landscape. Researchers anticipate extending CompreSSM's principles to further refine its effectiveness across various machine learning tasks.