System One Lite – typed decisions from a local LLM, with no generated tokens (github.com)

A new project, System One Lite, aims to transform local language models (LLMs) into efficient decision engines without relying on text generation. Instead of generating verbose textual outputs, System One Lite accepts an unstructured input state along with up to 64 typed questions, producing a straightforward probability distribution for each question while eliminating unnecessary token generation. By focusing on typed outputs, it streamlines the decision-making process for software applications, ensuring that results are constrained to predefined options without the risk of models fabricating responses. This innovation holds significant implications for the AI/ML community, particularly in enhancing the operational clarity between AI systems and code. System One Lite operates with minimal dependencies, running on Apple silicon hardware and using a standard open-weight model. Its design emphasizes reliability in decision-making by segregating the model's outputs from the application's logic. The model can only select from designated options, thus avoiding unpredictable behavior that arises from free-form text generation. Various practical applications for this model include routing customer support inquiries, scoring risks, and assessing leads, thereby providing a robust solution for environments where clarity and precision are paramount.