Validating LLM code edits when you can't run the code (velyr.io)

A new multi-layer validation system for code edits generated by large language models (LLMs) has been deployed, particularly for scenarios where the code cannot be executed. This system is essential because it allows an agent to make pull requests against production websites without traditional safeguards like build, tests, or type checks. Instead, it employs five distinct validation layers, each designed to mitigate risks without leaving any behind, ensuring that changes made to the codebase are "provably not-broken" before human review. The validation process begins by restricting which files can be edited to prevent unauthorized changes. It then anchors edits directly to actual file bytes to avoid transcription errors. Static checks only reject edits with provable issues, while comparative checks validate changes without requiring a complete understanding of the file format. Finally, an adversarial second model reviews the modifications to confirm that the proposed changes align with the model’s claims. This systematic approach not only increases trust in automated code editing but also enhances the overall safety and reliability of deploying AI-generated code changes in production environments.