Jev-serve: Run latest qwen3.8, other frozen LLM models, mlx-supported (github.com)

Jev-serve has launched a new endpoint that allows for rapid, typed probabilistic decision-making using large language models (LLMs) without generating text. This innovative approach significantly speeds up structured decision scoring, achieving a remarkable 34-fold increase in efficiency, reducing decision-making time from an average of 7.80 seconds to just 0.23 seconds. By reading the model's next-token logit distribution, jev-serve provides calibrated probabilities for various options, enabling precise outputs for three question types: choice, noun (binary yes/no), and score (ordinal levels). This advancement is particularly significant for the AI/ML community as it enhances the usability of LLMs in decision-making contexts, allowing for faster and more reliable structured outputs. Jev-serve runs on MLX, supporting direct inference on local models like Qwen3.8, while also being compatible with OpenAI APIs. The system guarantees schema consistency, minimizing issues like hallucinated outputs often seen with traditional text generation, thus ensuring that the probabilities generated are credible and well-calibrated. This tool could revolutionize industries needing quick, accurate decision-making, paving the way for more intelligent applications of LLM technology.