Show HN: Litmus – Specification testing for structured LLM outputs (github.com)

Litmus has been introduced as a powerful tool for conducting specification testing on structured outputs from large language models (LLMs). It enables users to define test cases that include input strings and expected JSON outputs, which can be executed against various LLM models via OpenRouter. This tool not only facilitates a comparison of accuracy but also assesses latency and throughput metrics, providing valuable insights into model performance. For example, tests conducted on OpenAI's GPT-4.1-nano and Mistral's Mistral-Nemo models showed both achieving 100% accuracy on specified tasks, with distinct latency and token efficiency metrics. The significance of Litmus lies in its ability to promote reliability and standardization in LLM outputs, which is critical as AI models become increasingly integrated into applications requiring structured data responses. By offering a detailed test report that includes pass/fail counts, latency percentiles, and a comparison table for different models, Litmus empowers developers and researchers to benchmark and enhance the performance of their LLM implementations. This step towards structured specification testing represents an important advancement in ensuring that AI systems deliver consistent and predictable results, aligning their outputs with user expectations and data standards.