Show HN: A new benchmark for testing LLMs for deterministic outputs (interfaze.ai)

A new benchmark named "SOB" has been introduced for evaluating large language models (LLMs) on their ability to produce deterministic outputs from various data formats, such as text, images, and audio. Existing benchmarks often fail to address real-world complexities by only measuring schema compliance or value correctness in isolation. The SOB framework evaluates structured output across three modalities using a rigorous scoring system that separates extraction capabilities from reasoning, ensuring more reliable assessments of models' performances. It reports seven metrics, including Value Accuracy and JSON Pass Rate, shedding light on the quality of outputs beyond simple validity checks. The significance of SOB lies in its potential to improve the reliability of LLM-generated structured data, which is critical in sectors like finance and healthcare, where inaccuracies can disrupt downstream systems. Findings from the benchmarking reveal a notable gap between high parsing rates and lower value accuracy, highlighting an opportunity for model development aimed at enhancing extraction fidelity. This initiative not only sets a new standard for benchmarking but also encourages developers to focus on producing models capable of delivering consistent and accurate data outputs, thereby paving the way for more effective integration of AI technologies in practical applications.