AgentRE-Bench: Can LLM Agents Reverse Engineer Malware? (www.agentre-bench.ai)

AgentRE-Bench is a new benchmarking framework designed to evaluate the capabilities of Large Language Model (LLM) agents in reverse engineering malware, specifically compiled ELF binaries targeting Linux/Unix. The framework challenges LLMs with complex tasks that require tracking and interpreting the outcomes of multiple tool calls (10–25), assessing their ability to identify command-and-control (C2) infrastructure, anti-analysis techniques, and communication protocols without human intervention. This goes beyond basic Q&A assessments, emphasizing real-world reasoning in sequence processing. This initiative is significant for the AI/ML community as it provides a structured method to measure LLM performance in security-related tasks, showcasing their potential and limitations in the field of cybersecurity. With levels ranging from identifying simple TCP reverse shells to complex metamorphic droppers employing advanced techniques like process hollowing and self-modifying code, AgentRE-Bench serves as a comprehensive tool for researchers. It provides a consistent scoring mechanism against a fixed ground truth, enabling reproducible results and facilitating the exploration of LLMs’ capabilities in recognizing and countering malware threats.