Turn any Llama-server into a jev system one endpoint (github.com)

A new open-source project called Verdict has been announced, designed to allow users to turn their existing Llama servers into a compatible decision-making endpoint that mirrors the functionality of Typesafe's hosted JEvil system. Verdict supports the same API wire protocol as JEvil, enabling developers to simply change the base URL in their existing JEvil clients to connect to their self-hosted solutions. This transition marks a significant move toward greater autonomy and control over AI models, as users can deploy models of their choice directly on their hardware, ensuring data privacy and reducing reliance on external services. Key technical features of Verdict include its ability to process typed questions and provide options without generating random text outputs, allowing for precise decision-making. By reading the probability distribution of options and providing confidence signals, Verdict delivers a clearer indication of action viability compared to traditional generative models. The implementation emphasizes accuracy and performance, as users can derive and measure option probabilities consistently, even on various hardware loads. With the capability to serve any chosen model and a stated focus on low-latency operations, Verdict not only enhances AI deployment flexibility but also challenges the norm of dependence on hosted AI solutions.