Limits and opportunities of AI reviewers: Reviewing the reviews of Nature papers (arxiv.org)

A recent study explored the effectiveness of AI reviewers in the context of scientific peer review, specifically analyzing reviews of papers published in Nature. Conducted by 45 expert scientists, the research involved a comprehensive review of both human-generated and AI-generated critiques of 82 Nature papers. The findings indicate that an AI reviewer, powered by GPT-5.2, surpassed the best human reviewer in delivering accurate criticisms, achieving a score of 60.0% compared to 48.2% for the top human critic. Additionally, AI reviewers unearthed 26% more issues than their human counterparts, demonstrating their potential to enhance the peer review process. However, the study also highlighted significant limitations. AI reviewers showed considerable overlap in their assessments (21%) compared to humans (3%), and they exhibited recurring weaknesses, such as insufficient specialized knowledge and an overly critical stance on minor points. The results suggest that while current AI reviewers can serve as valuable complements to human reviewers, they are not yet ready to replace them entirely. This research emphasizes the need for ongoing evaluation of AI capabilities in academic contexts and invites further exploration of how these tools can be effectively integrated into the peer review process.