Show HN: Portal2Bench – Can AI agents solve Portal 2? (kingstonyip.com)

Portal2Bench is an innovative project that tests various AI models' problem-solving abilities within the game Portal 2. Notably, models like GPT-6.1 Sol and GPT-6.1 Astra successfully solved 20 out of 26 test chambers, showcasing their high-level reasoning capabilities, while others such as Fable 5.1 and Sonnet 5.5 solved only a few. The framework allows AI agents to interact with the game by sending inputs and receiving feedback in the form of screenshots, all while adhering to a strict runtime and budget constraint for each chamber. This initiative is significant for the AI/ML community as it pushes the boundaries of how AI agents can engage with complex environments that require spatial reasoning and strategic planning. By quantifying their performance, with median costs per chamber ranging from $0.74 to $13.12, the project provides valuable insights into the economic feasibility of deploying such models in gaming and problem-solving scenarios. Moreover, the methodology of running each model independently under controlled conditions enhances understanding of their capabilities and limitations without needing direct external resources, underscoring a new frontier in AI interaction within virtual environments.