LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques (github.com)

The newly launched "LoRA Speedrun" introduces a public leaderboard aimed at fine-tuning the Qwen2.5-1.5B model using techniques that prioritize wall-clock time over traditional metrics like FLOPs. Participants can compete to achieve a minimum accuracy of 57% on the GSM8K dataset using a framework that standardizes the environment with frozen hardware and task parameters. The current record stands at 6 minutes and 5 seconds, set by user @Saivineeth147, employing advanced strategies such as sequence packing and completion-only loss masking within just two epochs. This initiative is significant for the AI/ML community as it offers a structured, head-to-head format for comparing various parameter-efficient fine-tuning methods, something lacking in existing literature where results are often reported across different models and datasets. By ensuring that all trials are conducted under identical conditions in a Modal sandbox, the competition not only fosters innovation in fine-tuning practices but also generates verifiable results that can impact real-world applications of LoRA and its derivatives. This approach emphasizes practical, real-world performance, making it a pivotal step in the evolving landscape of model fine-tuning.