Turning Historical Incidents into AI Insights (blog.paypay.ne.jp)

PayPay announced GBB RiskBot, an automated code-review assistant that mines the company’s historical incident and postmortem data to flag risks in pull requests. The system ingests incident reports via a GitHub Actions cron job, preprocesses and normalizes them, then creates OpenAI embeddings (text-embedding-ada-002 today) which are indexed in a VectorDB (ChromaDB chosen for low cost and easy POC setup). On each PR, changes and metadata are vectorized (with a 1,000-character cap per item for performance and cost control), a cosine-similarity search retrieves top-K similar incidents, and a RAG-style prompt uses gpt-4o-mini to synthesize a GitHub comment. The architecture leans on retrieval for pattern matching, so the LLM’s role is mainly formatting and contextualizing known failure modes rather than heavy reasoning. This approach automates cross-team knowledge transfer, limits recurrence of known failure patterns, and scales code review as PayPay’s codebase grows. Operational costs are minimal (example: database init for 47 incidents ≈ $0.00185; one-file PR analysis ≈ $0.00035; July’s ~380 runs cost ~$0.59), while success is tracked via tiered metrics (issue-detection rate, incident coverage, developer feedback via GitHub reactions, and long-term incident-rate reduction). Planned improvements include upgrading embeddings to text-embedding-3-large, adding a rerank step to cut false positives, and experimenting with alternatives to RAG to boost precision.