OpenReviewer: A Specialized LLM for Generating Critical Scientific Paper Reviews (aclanthology.org)

OpenReviewer, an innovative open-source system designed to generate high-quality peer reviews for machine learning and AI conference papers, has been unveiled. At the heart of this tool is the Llama-OpenReviewer-8B, an 8 billion parameter language model meticulously fine-tuned using a dataset of 79,000 expert reviews from prestigious conferences. This system autonomously extracts critical elements from PDF submissions, such as text, equations, and tables, and crafts structured reviews in accordance with specific conference guidelines. Evaluation of OpenReviewer against 400 test papers revealed it produces significantly more critical and realistic reviews compared to widely used general-purpose models like GPT-4 and Claude-3.5, which often skew towards overly positive feedback. The introduction of OpenReviewer is significant for the AI/ML community as it addresses the longstanding need for reliable pre-submission review tools that mirror the rigor of human assessment. Its ability to deliver timely, constructive feedback not only aids authors in refining their manuscripts but also enhances the overall quality of academic discourse. Importantly, OpenReviewer is not intended to supplant human reviewers but rather to serve as a valuable resource in the peer review process. The tool is accessible online and available as open-source software, providing researchers with an innovative means to elevate their work before submission.