Enhancing Transformer-Based Rerankers with Synthetic Data and LLM Supervision (arxiv.org)

Researchers propose a practical pipeline that uses Large Language Models (LLMs) to create and label synthetic training data for transformer-based rerankers, removing the need for scarce human-labeled query–document pairs. The method has two LLM-driven stages: (1) generate domain-specific queries from an unlabeled corpus, and (2) use an LLM classifier to assign positive and hard-negative labels. The synthetic pairs train a smaller, efficient transformer reranker with contrastive learning using Localized Contrastive Estimation (LCE) loss. Applied to the MedQuAD dataset, this strategy markedly improves in-domain reranking and generalizes well to out-of-domain tasks. This approach is significant because it leverages LLMs’ semantic and reasoning strengths for offline data creation rather than costly online inference, enabling low-latency deployment of compact rerankers with near-LLM performance. Key technical implications include automated hard-negative mining via LLM supervision, effective contrastive fine-tuning with LCE, and scalable domain adaptation without manual annotation. Practical benefits target specialized search systems (e.g., medical Q&A), offering better relevance at much lower serving cost; however, model and labeling quality remain tied to the LLM used for synthetic supervision, so careful LLM selection and bias monitoring are important in production.