MMTB: Evaluating Terminal Agents on Multimedia-File Tasks (arxiv.org)

A new benchmark, MultiMedia-TerminalBench (MMTB), has been introduced to evaluate terminal agents' performance on tasks involving multimedia files, such as audio and video. Unlike traditional benchmarks that focus primarily on text and code, MMTB includes 105 tasks across five meta-categories, specifically designed to challenge AI agents in manipulating and interpreting multimedia content. This shift is significant because many real-world workflows now demand automation not just in textual contexts but also in areas requiring auditory and visual processing. Accompanying MMTB is the Terminus-MM, a multimedia harness that enhances the capabilities of existing terminal agents like Terminus-KIRA by integrating audio and video perception. This combination allows researchers to systematically study how access to different forms of multimedia influences task completion and what evidence agents leverage to develop executable workflows. The release of MMTB and Terminus-MM is poised to advance the understanding of multimedia's role in AI/ML, paving the way for more sophisticated automation in complex, multimedia-rich environments.