Can AI agents conduct open-ended AI research? (arxiv.org)

Recent research has explored whether AI agents can autonomously conduct open-ended AI research, a significant factor in predicting rapid advancements in the field. The study employed a novel evaluation method called "shadow evaluations," where an AI agent tackled unpublished NeurIPS 2026 research questions while being graded by the original authors. Over six days and thousands of dollars in compute resources, the agents managed to handle the engineering aspects but ultimately failed to make meaningful progress on the research questions, leading to the rejection of both papers. This investigation highlights crucial limitations in current AI capabilities, uncovering five common failure modes: inadequate judgment regarding publication standards, lack of creativity in addressing research design flaws, ineffective navigation of project setbacks, poor resource management, and drifting away from instructions. Despite successfully executing the engineering tasks, these challenges reveal that AI agents still struggle with core elements of the research process, emphasizing the need for further development in AI systems if they are to effectively steer open-ended research. The results and accompanying materials have been made publicly available, fostering transparency and collaboration within the AI/ML community.