Open alternative to TypeSafe's Jev, running locally on your own GPU (github.com)

Open-source developers have introduced an alternative to TypeSafe's Jev, called the "open-alternative-jev," which allows users to run typed, calibrated decision-making processes on their local GPUs using any open-weights large language model (LLM). This Python package, accessible through the import name "so1" ("System One"), processes questions without generating text, focusing instead on deriving answers from the next-token distribution of provided options. The benchmark results show impressive accuracy, with the open alternative achieving up to 92.9% accuracy on RACE-H tasks while significantly improving processing speed by packing question sequences, thereby reducing token usage. This development is significant for the AI/ML community as it democratizes access to advanced decision-making capabilities, making it possible for researchers and developers to leverage sophisticated LLMs without relying on proprietary solutions. The implementation provides essential features such as temperature scaling for calibration and two operational modes: "packed" for efficiency with shared states and "separate" for independent question processing. By harnessing the power of existing LLMs, this library not only enhances throughput but also expands the potential for real-time applications in various domains, offering a compelling solution for those interested in deploying AI models locally.