Do you put rules or examples in your LLM context? (blog.getcassis.com)

Recent research has explored how the inclusion of examples versus general rules in Large Language Model (LLM) contexts influences their performance on data analytics tasks. The study introduced the concept of "derivation distance," which quantifies the effort an AI model must exert to transform given context into an answer. Findings suggest that using specific examples considerably reduces this distance and enhances accuracy, especially on complex questions, as demonstrated by notable performance boosts for models like Haiku and Sonnet when provided with example libraries. The significance of this research lies in its implications for optimizing AI training approaches. By emphasizing derivation distance and the benefits of structured, contextual examples, the study provides a framework for enhancing LLM capabilities across various applications. Key technical insights include the observation that smaller models gain significantly from examples, while larger models like Opus perform well with less contextual burden. The comprehensive results, supported by reproducible experiments, pave the way for future advancements in AI performance by guiding how context is structured around problem-solving in specific domains.