AndroAndroid Bench 2.0: Pushing the frontier with challenging long-horizon tasks (android-developers.googleblog.com)

Android Bench 2.0 has been launched, introducing a set of long-horizon tasks (LHT) designed to challenge large language models (LLMs) with complex engineering tasks that can take days or even weeks to complete. This major upgrade moves beyond the initial focus on smaller changes and bug fixes, reflecting the evolving capabilities of AI assistance in software development. The new framework aligns with the Harbor framework and implements continuous scoring, focusing on evaluation metrics like functionality and visual fidelity, rather than a binary pass/fail system. Currently, the highest pass rate for these LHTs hovers around 28%, indicating the significant complexity involved compared to the approximately 91% pass rate for earlier tasks. This development is significant for the AI/ML community as it enhances the evaluation of AI models, enabling developers to better understand the strengths and weaknesses of different models and agents in real-world scenarios. With benchmarks that assess how well models assist with tasks such as upgrading dependencies and app migrations, Android Bench 2.0 provides practical insights into AI’s effectiveness in coding. Additionally, the inclusion of agent evaluations allows for a deeper exploration of how different models integrate with developer workflows. As the landscape of AI continues to evolve, this tool aims to empower researchers and developers alike, facilitating the selection of optimal AI solutions for Android development projects.