Can My Puzzle Write-Ups 'Lift' Cheap LLMs? (matthewritch.com)

A recent experiment by a tech enthusiast aimed to enhance the problem-solving capabilities of cheaper language models (LLMs) using detailed puzzle solution write-ups. By leveraging the Inspect eval harness within a Jupyter-style environment, the author meticulously tested models like Haiku 4.5 and Sonnet 5 on a series of puzzles, incrementally providing them with sections of their solution posts. The study focused on determining the minimum information needed from these write-ups to enable the models to successfully tackle puzzles—crucial for understanding how to maximize lower-cost LLMs effectively. The tests revealed that, despite the additional guidance, Haiku 4.5 struggled significantly, while Sonnet 5 showed greater promise in solving the puzzles when given more context. Notably, the author found that adjustments to the testing environment—such as preserving the Python state between calls and increasing memory allocations—were pivotal in improving the models' performance. This exploration highlights the broader implications for the AI/ML community regarding the scalability of cheaper models, suggesting that with tailored instructions and strategic modifications, even less powerful LLMs could be equipped to tackle complex tasks more effectively.