The Pi0-Fast Libero-10 Baseline Is 25 Points Low (topicqueue.substack.com)

Recent analysis reveals that the pi0-FAST model's reported performance on the LIBERO-10 benchmark is significantly misguided. Initially cited as a 60.2% success rate, it has been corrected to an impressive 85% after addressing a misconfiguration that caused a duplicate transform during fine-tuning. By disabling the erroneous transform, the model's success rate surged from 60.5% to 84.7%, demonstrating that many current benchmarks erroneously cite outdated performance numbers. This revelation is crucial for the AI/ML community as many researchers rely on these baseline figures for comparative evaluations. The implications suggest that existing models underperform on LIBERO-10 if not correctly configured; hence, the true potential of pi0-FAST is substantially higher than previously thought. In addition to the pi0-FAST revelations, a new dataset drop by Unitree includes 60 humanoid manipulation datasets covering various real-world tasks, although none of them are licensed for public use. This influx of data represents a major contribution to the robotics field, but the lack of licensing could hinder its immediate applicability. In contrast, the MLPerf v6.1 benchmark highlighted a 5.7x improvement in per-accelerator inference capabilities, emphasizing the rapid advancements in AI hardware and models, as the community continues to push boundaries in performance metrics.