Show HN: I built a small audit layer for LLM-as-judge decisions (github.com)

A PhD student has developed CMG, a small audit layer designed to improve the reliability and transparency of language models (LLM) acting as judges in evaluation tasks. This tool ensures that every decision made by an LLM is backed by explicit claims directly tied to the evidence provided, addressing a significant issue in AI evaluations where biases and inconsistencies can occur. CMG identifies potential pitfalls by flagging cases where judges fail to cite evidence or adequately address evaluation criteria, thus enhancing the trustworthiness of LLM outputs. This innovation is particularly significant as it offers a systematic approach to auditing LLM decisions, which is crucial for researchers relying on these models for grading and assessment tasks. CMG does not attempt to eliminate inherent biases but makes them more visible, allowing users to pinpoint verdicts that lack sufficient justification. With features such as a web dashboard for tracking evaluations and a focus on ensuring all criteria are correctly covered, CMG provides a robust framework for monitoring LLM performance, thereby fostering greater accountability in AI system evaluations.