Bag of Decisions Reranker (softwaredoug.com)

A novel approach to search relevance has been proposed, termed the "Bag of Decisions Reranker." Instead of treating search queries as a mere "bag of terms," this method utilizes a Large Language Model (LLM) to generate a rubric of yes/no questions tailored to assess the relevance of documents for specific queries. For instance, for the query "desk for kids," relevant questions might include whether the document discusses child-sized desks or mentions safety features. Then, using a model named Jev, the approach calculates the relevance score by summing the probabilities of each question being answered affirmatively. This technique is significant for the AI/ML community as it represents a shift towards more sophisticated query handling in information retrieval systems. Early experiments conducted on e-commerce datasets, such as Wayfair WANDS and Amazon ESCI, showed that the Bag of Decisions Reranker outperformed both traditional BM25 and Jev reranking methods in terms of mean and median NDCG scores. The implications suggest potential development of customized decision models for specific domains, marking a promising direction for improving the accuracy and relevancy of search results through enhanced query feature generation.