Which one is more important: more parameters or more computation? (2021) (parl.ai)

Recent research has highlighted the importance of distinguishing between model parameters and computational resources in deep learning, challenging the traditional view that equates the two. Two innovative approaches have been introduced: Hash Layers and Staircase Attention models. Hash Layers leverage a simple hashing mechanism to allocate model parameters across sparse mixture-of-expert architectures, enabling larger models to operate efficiently with reduced computational load. This method has shown significant performance improvements in language tasks, outperforming conventional learning-based models, while drastically reducing the number of active parameters per input. On the other hand, Staircase Attention models propose a new architecture that separates the concepts of computation and parameters, allowing increased computation without adding parameters. By stacking Transformers in a recurrent manner, these models enhance performance in scenarios where maintaining internal state is crucial. Importantly, a combination of both methods appears to unlock even greater potential, underscoring the need for a nuanced approach to model design. This research opens up fresh avenues for the AI/ML community, suggesting that optimizing for computation and parameters independently can lead to powerful, more resource-efficient deep learning models.