The Sparsely-Gated Mixture-of-Experts Layer (2017) [pdf] (arxiv.org)

A recent advancement in neural network architecture has been introduced with the Sparsely-Gated Mixture-of-Experts layer (MoE), which allows for conditional computation to enhance model capacity dramatically—over 1000 times—while only slightly impacting computational efficiency. This layer utilizes a mechanism where a gating network activates a sparse selection of up to thousands of feed-forward sub-networks, enabling each instance of data to harness the power of specialized experts. This innovation is particularly impactful for tasks requiring immense model capacity, such as language modeling and machine translation. The implications of this development are significant for the AI/ML community, as it enables the creation of models with up to 137 billion parameters that achieve superior performance on large datasets without a proportional increase in resources needed. By applying the MoE in convolutional layers between stacked LSTM networks, researchers have demonstrated that these new architectures not only outperform existing state-of-the-art models but do so more efficiently, highlighting the potential for even larger and more capable neural networks in future applications.