The Lottery Ticket Hypothesis: finding sparse trainable NNs with 90% less params (arxiv.org)

Researchers have made a significant advancement in neural network training with the introduction of the "lottery ticket hypothesis." This hypothesis posits that dense, randomly-initialized neural networks contain sparse subnetworks—referred to as "winning tickets"—that can be trained effectively in isolation, achieving test accuracies comparable to their larger counterparts. Notably, these winning tickets can be as small as 10-20% of the size of fully-connected and convolutional architectures, enabling a pruning process that reduces parameters by over 90% while maintaining performance. The implications of this work are profound for the AI/ML community. By identifying winning tickets that exhibit effective training performance due to their advantageous initial weights, researchers can not only alleviate storage and computational burdens but also enhance training efficiency. The study offers a novel algorithm to locate these subnetworks, demonstrating that they often learn faster and achieve higher accuracy than the original model when trained. This contribution may pave the way for developing more efficient neural networks, optimizing both training times and resource usage in various applications.