UniPool: A Globally Shared Expert Pool for Mixture-of-Experts (arxiv.org)

Researchers have introduced UniPool, a novel Mixture-of-Experts (MoE) architecture that revolutionizes the conventional expert allocation method in transformer models. Traditionally, each transformer layer is assigned a distinct set of experts, leading to an increase in the number of parameters as depth increases. However, UniPool proposes a paradigm shift by establishing a globally shared expert pool, allowing multiple layers to access a centralized set of experts. This approach not only reduces the parameter growth rate but also maintains or improves model performance. By leveraging a pool-level auxiliary loss to balance expert usage and implementing a NormRouter for effective routing, UniPool demonstrates significant enhancements in validation loss and perplexity across various model sizes, outperforming traditional layer-wise MoE configurations. The significance of UniPool lies in its efficiency and flexibility. By enabling expert parameters to grow sublinearly rather than linearly with depth, it challenges existing assumptions in MoE architecture, potentially leading to more compact yet effective models. This opens new avenues for model scaling and optimization, allowing for superior performance without a proportional increase in computational resources. The successful implementation of UniPool offers promising implications for future AI/ML applications, setting a new standard in the development of scalable and efficient transformer-based systems.