parameter-golf (github.com)

OpenAI has launched the Parameter Golf challenge, inviting participants to create the best language model constrained to a 16MB size and trained within 10 minutes on an 8xH100s setup. This competition, inspired by the NanoGPT Speedrunning challenge, encourages innovation in model architecture and compression techniques, such as parameter tying and low-rank training. Participants will evaluate their models using the FineWeb validation set, with submissions open from March 18 to April 30. This initiative is significant for the AI/ML community as it pushes researchers to explore creative approaches in parameter-efficient training while optimizing for performance metrics. By limiting the resource constraints, OpenAI aims to catalyze advancements in model efficiency and performance, potentially yielding breakthroughs in areas such as test-time training and novel tokenizers. OpenAI is also providing $1,000,000 in compute credits to lower barriers for participation, thus fostering an inclusive environment for innovative research.