We review every PR with a 27B model on self-hosted infra (varoa.net)

A new automated review system leveraging a self-hosted 27B model has been implemented to streamline the process of pull requests (PRs) for a major software project. In its initial month, the system provided feedback on 75% of PRs within just eight minutes, significantly improving efficiency in software delivery. By using a bespoke architecture that separates deterministic tasks from complex language model inputs, the team aims to maintain consistency and quality without overwhelming engineers with feedback. The system relies on structured data exchanges, with the LLM focusing solely on understanding context-rich code diffs, thus preserving valuable resources and reducing costs by minimizing reliance on expensive hosted models. This development is significant for the AI/ML community as it illustrates a practical application of language models in software engineering, emphasizing the role of automation in enhancing human productivity. Key technical components include an observer for GitHub events, a scheduler for task management, and the use of the vLLM inference provider to manage costs and resource utilization. Additionally, the system prioritizes user interaction dynamics, allowing for seamless integration with human workflows rather than outright replacement, thus addressing ongoing concerns about knowledge transfer and collective ownership within engineering teams. The experiments conducted also emphasize iterative improvement based on detailed metrics on feedback relevance and model performance, paving the way for more refined development processes.