Pretraining Latent Information Feedback Transformers with Teacher Supervision (arxiv.org)

Researchers have introduced the Latent Information Feedback Transformer (LIFT), a novel architecture aimed at enhancing transformer language models by enabling better information flow between layers during pretraining. Traditional transformers process information in a feed-forward manner, relying heavily on the output token for intermediate states, which limits their ability to learn effectively. LIFT addresses this bottleneck by transforming the training process into a teacher-forced prediction problem, allowing for the propagation of rich latent states derived from pretrained models, thereby optimizing both next-token and next-state predictions. This advancement is significant for the AI/ML community as it demonstrates that deep-to-shallow feedback mechanisms can significantly improve language modeling and task performance without increasing computational costs. In experiments involving models with various parameter sizes, LIFT not only outperformed standard transformers on language and reasoning tasks but also showed promise in efficiency, requiring less data to achieve superior results. By leveraging teacher supervision for state prediction, LIFT opens doors for more effective and scalable pretraining methods in AI, pushing the boundaries of current language model capabilities.