Enabling small language models to solve complex reasoning tasks (news.mit.edu)

Researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a novel framework called "Distributional Constraints by Inference Programming with Language Models" (DisCIPL) that enables small language models (LMs) to effectively tackle complex reasoning tasks traditionally dominated by larger models. By leveraging a larger "boss" LLM to plan and instruct multiple smaller follower models, DisCIPL improves the accuracy and efficiency of responses for tasks such as writing constrained text, generating grocery lists, and planning itineraries. This method not only produces results comparable to leading models like OpenAI's GPT-4o but also offers significant cost savings and faster processing times, with the potential for scalability by combining numerous small LMs. The significance of DisCIPL lies in its innovative approach to enhance the efficiency of language models while reducing their energy consumption — a crucial factor as demand for powerful LMs grows. By employing a programming language called "LLaMPPL" to encode specific rules, the system allows LMs to collaborate intelligently, optimizing both output quality and computation costs. Early experiments revealed that DisCIPL outperformed baseline models in various tasks, demonstrating its potential for broader applications in areas requiring nuanced reasoning and adherence to constraints. Future research will explore further enhancements, including recursive modeling capabilities and the application of DisCIPL to complex mathematical reasoning.