LLM Rerankers for RAG: A Practical Guide (fin.ai)

A practical guide from ainews247’s team explains how they built an LLM-based reranker to improve retrieval-augmented generation (RAG) grounding, deployed it in production for their Fin and Copilot agents, and open-sourced the exact prompt. They found open-source cross-encoders fast but insufficient in quality, so they used an LLM to score top-K passages (K=40). In A/B tests the LLM reranker raised resolution and assistance rates (Fin: statistically significant uplift; Copilot: +3pp assistance, +2pp answer rate, +63% citations of public/internal articles) and acted as a teacher to train a lower-latency custom reranker. Key engineering moves made it practical: they chose pointwise scoring for clear numeric outputs, drastically reduced output tokens (switched to compact Dict format, removed spaces, thresholded to omit scores under 5—cutting output tokens ≈28% then ~50% latency), and parallelized scoring by splitting passages into N batches with round-robin assignment to avoid positional bias (e.g., N=4). Results: end-to-end added latency fell from ~+5s to <1s (P50 +0.9s) and costs dropped ~8x. They merge scored shards, use a BGE cross-encoder to break ties or fill missing shards, and mitigate timeout risk with per-call timeouts and fallbacks. Trade-offs remain—extra latency and orchestration complexity—so they distilled a strict scoring rubric and JSON-only prompt (scores 5–10) to keep consistency and reliability at scale.