Guidance injection: reliable instructions for local LLMs (samihonkonen.com)

A new approach called guidance injection has been introduced to address a common limitation in smaller, local language models (LLMs): their tendency to forget instructions over the course of a conversation. Typically, instructions are embedded in a system prompt, but as interactions progress, these models can fail to adhere to them. Guidance injection changes this by delivering instructions at the precise moment they are needed, rather than relying on memory. By attaching instructions to specific tools, the system checks for applicable guidance before executing operations, ensuring that the model remains aligned with user intentions. This methodology is particularly significant for the AI/ML community as it enhances the reliability of smaller models, which often struggle with attention limitations. Instead of a bulky system prompt filled with every rule, guidance is delivered contextually, improving both efficiency and performance during specific interactions. The implication is twofold: it allows for a cleaner system prompt and provides a mechanism that caters to the operational needs of smaller models, which benefits their functionality in real-world applications where maintaining context is critical. This innovation could lead to more effective use of memory in LLMs, making them more reliable for applications that require sustained user engagement.