Lessons from a weekend building local AI workflows (stefano.petrilli.xyz)

A recent exploration into multi-agent workflows has unveiled crucial lessons for the AI and machine learning community, particularly concerning the limitations of current models. An experiment aimed at creating a multi-agent video editor, which identifies and cuts unnecessary content from lengthy videos, faced significant challenges. Among the key findings were the "lost-in-the-middle" phenomenon, where language models struggle to effectively retrieve information from the middle of their context, often favoring the beginning or end. This issue resulted in the editor overemphasizing introductory content rather than the more substantial middle sections. Implementing a new agent to identify core messages significantly improved the output. Additionally, issues of bias compounding were noted when trying to achieve consensus between an editor and reviewer, as both utilized the same model, leading to premature agreement on flawed outputs. Switching to different model families for these roles helped counteract this effect. The research also highlighted that commonly accepted models like Whisper have limitations, particularly in timestamp accuracy and natural sentence segmentation. Ultimately, using alternatives like Vosk improved transcription quality, demonstrating that less popular models can outperform well-known solutions in specific contexts. These insights underscore the need for continuous adaptation and evaluation of AI workflows to enhance performance in practical applications.