New in Llama.cpp: Decision Models (huggingface.co)

Llama.cpp has introduced a new feature called Decision Models, which allows users to send a state along with typed questions to receive probabilities for each option in a single forward pass instead of generating text. This innovation follows the System One format from TypeSafe's Jev model, simplifying integration for existing clients with just a new base URL. Decision Models are particularly significant as they optimize how AI can facilitate decision-making processes, such as routing customer requests or assessing urgency without the overhead of generating full responses token by token. Key technical capabilities include various model sizes ranging from 144M to 27B parameters and their ability to handle different question types, including choice, scoring, and yes/no formats. The models can also accept multiple inputs, such as text and images, which broadens their application range—ideal for tasks like classifying documents or enhancing customer support systems. This update underscores a shift towards more efficient AI applications, enabling faster and more precise decision-making, which can help businesses streamline operations and improve user experiences.