Requential Coding <1 bit compression with better generalization (github.com)

A groundbreaking method called Requential Coding has been introduced, which achieves less than one bit compression while significantly enhancing the generalization capabilities of generative models. This technique innovatively compresses a generative model by utilizing a training process that derives its samples from self-generated data using a two-model framework: a student model that creates its training batches and a teacher model that selects these batches based on real data. The uniqueness of this approach lies in its use of relative entropy coding, which yields a cumulative teacher-student Kullback-Leibler divergence that's independent of the model's parameters and data entropy. This development is significant for the AI/ML community as it pushes the boundaries of model compression techniques, potentially lowering computational costs and enhancing model performance in practical applications. With the added capability of generating synthetic data efficiently across multiple devices, the Requential Coding method is poised to influence future training paradigms and efficiency metrics. Key technical features include the detailed logging of performance metrics and experiments run on TPUs and GPUs, providing an accessible and powerful toolkit for researchers and developers aiming to fine-tune generative models.