A detailed review after 30 days of using CodeRabbit for PRs (lycheeorg.dev)

An independent 30-day trial of CodeRabbit for reviewing GitHub pull requests found the tool to be a practical, high-value complement to human reviews. Setup is simple—OAuth via GitHub and repo selection—and the bot auto-edits PR descriptions, performs multi-minute analyses, posts structured “request changes” comments, and incrementally re-analyzes after commits or replies; once all comments are resolved it can auto-approve the PR. Over 28 PRs (32,784 lines added, 4,768 removed, 693 files), CodeRabbit flagged 290 issues, including critical security findings (Zip Slip, IDOR/cross-user access, validation/logic bugs) and many quality improvements that would have led to real bugs. The author categorized findings: 15% useless, 13% wrong assumptions, 21% nitpicking, 13% thoughtful prompts, 35% quality improvements, and 3% security/critical—meaning ~72% were relevant and about 51/72 (~71%) materially improved code quality. Limitations include a slightly sluggish UI and open-source plan rate limits (200 files/hour; review cadence: 3 back-to-back then 2/hr; conversation limits: 25 back-to-back then 50/hr). The reviewer concludes CodeRabbit doesn’t replace humans but meaningfully reduces missed issues and is recommended for open-source maintainers, noting features like suggested code changes and IDE prompts exist but weren’t evaluated.