Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090 (www.gilesthomas.com)

A developer has successfully extended Sebastian Raschka's GPT-2 by implementing a Mixture-of-Experts (MoE) model trained from scratch on an RTX 3090 GPU. This novel model comprises 446 million parameters with 220 million active parameters, essentially operating as a GPT-2 small enhanced with six experts—activating two per token. This approach enhances inference speed while maintaining the knowledge of larger models, making it significant in AI/ML for optimizing resource usage during language model training. Despite taking four times longer to train, this custom MoE model outperformed previous attempts in loss metrics, demonstrating promising performance in instruction fine-tuning tests. The implementation process involved integrating a router for efficiently determining which experts to activate based on incoming context vectors, enhancing learning capabilities without the overhead of running all experts simultaneously. Through the insights gained from classical papers on MoE architectures, the developer was able to create a model that aligns with existing frameworks while providing a practical understanding of MoEs in transformer-based architectures. Ultimately, this work showcases the potential of MoE systems in improving model efficiency and effectiveness, encouraging further experimentation in the AI/ML community.