Open benchmark of AI decision APIs vs. XGBoost for fraud triage (github.com)

A new vendor-neutral research benchmark has been introduced to evaluate specialized AI decision models against the conventional XGBoost algorithm in the context of payment-fraud ranking. This study compares the effectiveness of various models—including zero-shot decision models like TypeSafe Jev and OpenAI Decisions—by assessing how well they rank fraudulent transactions for review by analysts. Key metrics include average precision, recall rates at the top 1% and 5%, and fraud count within the top-ranked cases. This methodology sheds light on the cost and latency involved in fraud detection, focusing on the practical question of which models yield the highest detection rates. This benchmark holds significance for the AI/ML community as it emphasizes the need for models that not only perform well in isolation but also provide actionable insights for analysts who can only review a small fraction of flagged cases. The structured experimentation allows for offline testing while ensuring consistent input formats across different algorithms, facilitating fair comparisons. Furthermore, it highlights the emerging potential of zero-shot decision models in real-world applications, hinting at future shifts in the way fraud detection systems may evolve. By providing an open-source framework, the project fosters collaboration and innovation within the field, potentially leading to more robust solutions against payment fraud.