Dispersion loss counteracts embedding condensation in small language models (chenliu-1996.github.io)

Recent research highlights a geometric phenomenon called "embedding condensation" that affects small language models (LMs), causing their token embeddings to collapse into a narrow cone in high-dimensional space, leading to reduced expressivity. This effect is more pronounced in smaller models compared to their larger counterparts, which better maintain diverse representation capabilities. Notably, this condensation occurs early in model training and is not alleviated through knowledge distillation from larger models. To address this issue, the researchers introduce a new training objective known as "dispersion loss," aimed at counteracting embedding condensation and improving the representational qualities of smaller LMs. The significance of this work lies in its potential to enhance the performance of smaller language models without increasing their parameters. By implementing dispersion loss, smaller models can achieve representational qualities that are closer to those of larger models, thus narrowing the performance gap. This research opens up intriguing avenues for future exploration in the area of model training, suggesting that effective organization of information within latent representations may be just as critical to model performance as the sheer number of parameters.