Open-jev: One-pass option scoring with Gemma 3 4B, similar to jev (github.com)

Open-jev, a new AI scoring model inspired by the jev framework, has been announced, utilizing the Gemma 3 4B model to perform one-pass option scoring on Apple silicon devices. This independent reimplementation allows for efficient scoring of actions in real time, where the model evaluates a set of pre-written options based on a given context in a single padded forward pass. By using a local cache system and avoiding the need for decoding, Open-jev reduces the latency to approximately 90 milliseconds per scoring request—making it significantly faster and more efficient than previous methods. This development is significant for the AI/ML community as it enhances the capability to rank multiple options quickly and accurately, which is vital for applications like customer service automation and dynamic decision-making systems. The ability to assess various action options in one go, combined with techniques like softmax for probability distribution, opens new avenues for real-time AI applications. Key technical features include support for local model deployment on macOS using Metal, a streamlined API for integration, and compatibility with synthetic test data for benchmarking performance. The open-source nature of the project also encourages further innovation and collaboration within the AI community.