Low-cost, evasive malware for benchmarking reverse engineering capabilities (malwarebench.com)

MalwareBench has launched as the first end-to-end, cross-platform benchmark for evaluating AI capabilities in malware reverse engineering. This innovative framework assesses models' performance through realistic tasks involving obfuscated malware binaries across various operating systems, including Windows, Linux, macOS, and Android. Each case challenges AI systems to deliver both a behavioral analysis and a reusable artifact extractor, simulating the comprehensive analysis human analysts perform during incident responses. Unlike prior benchmarks that rely on synthetic tasks or known malware samples, MalwareBench employs functional, real-world malware that has been deliberately designed to evade detection by conventional defenses. The significance of MalwareBench lies in its potential to elevate AI models' effectiveness in identifying and analyzing novel malware types, which has substantial economic implications for cybersecurity. By emphasizing real task scenarios and long-horizon workflows, the benchmark tests generalization capabilities in models—integral for handling unseen malware variants. Initial evaluations suggest that even top-performing models, such as GPT-6.1 Sol, achieve only 7.6% effectiveness, indicating a critical opportunity for advancements in the field. Thus, MalwareBench serves not only as a measurement tool for AI progress but also highlights the urgent need for innovation in malware analysis to protect against emerging threats.