Prompter – Compare and benchmark Ollama models side-by-side in your terminal (github.com)

Prompter, a new terminal-based tool for the Ollama framework, has been launched, enabling users to compare, benchmark, and evaluate multiple models simultaneously with zero dependencies and a single-file setup. This innovative tool streamlines the evaluation process by allowing users to run the same prompt across different models and view the responses side by side, enhancing the analytical capabilities available to AI/ML developers and researchers. Prompter features structured evaluation modes including self-review loops, multi-model debates, and adversarial interrogations, facilitating deeper insights into model performance and reasoning. The significance of Prompter lies in its ability to support comprehensive evaluations that go beyond simple comparisons. By running 20 standardized capability tests—covering tasks like arithmetic, web searches, and file reading—Prompter provides robust markdown reports detailing model performance metrics. This makes it an invaluable resource for AI practitioners seeking to fine-tune models before deployment or researchers analyzing varied model behaviors in a controlled environment. The tool positions itself as a practical solution between lightweight usage and heavier evaluation frameworks, streamlining the workflow for anyone working with local LLMs.