Late-interaction rerank made our F1 worse, not better – a negative result (sverklo.com)

In a recent experiment, researchers from Sverklo evaluated a "poor-man's late-interaction rerank" method and found it to be detrimental, reducing their F1 score from 0.5847 to 0.5551 when tested with a dataset of 120 tasks. The reranking approach utilized a MiniLM-L6-v2 model, which was not adequately trained for the task at hand, leading to a decline in performance on definition lookups—a core functionality where exact matches are crucial. This drop highlights the model's propensity to prioritize semantic similarity over precise matches, which is counterproductive for queries based on exact symbol names. This negative result underscores the importance of aligning the right model with specific retrieval tasks, suggesting that rerankers can worsen outcomes when the primary ranker's accuracy is already high. It also emphasizes the necessity of rigorous benchmarking practices during experimentation to ensure valid results. Moving forward, Sverklo aims to reassess their methodology by implementing a proper ColBERT v2 model in hopes of achieving a substantial F1 lift while maintaining low latency, reflecting a commitment to transparency in the AI/ML community by sharing both successes and failures.