Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model (github.com)

A new AI model called Needle has been introduced, distilling the Gemini 3.1 architecture into a more compact 26 million parameter "Simple Attention Network." This model is designed for ease of use, allowing developers to finetune it locally on their Mac or PC. In operational terms, Needle achieves impressive speeds of 6000 tokens per second during prefill and 1200 tokens per second for decoding, showcasing its potential for efficiency. Key technical specifications include a dimensionality of 512, the use of 8 heads with 4 kilovectors, and BPE encoding with 8192 tokens. The significance of Needle lies in its positioning as an accessible entry point for personal AI applications on consumer devices such as smartphones and smartwatches. It outperforms larger models like FunctionGemma-270m and Qwen-0.6B in single-shot function calls, although it is noted that these larger models may still excel in conversational contexts due to their greater capacity. The open-source nature of the model allows developers to experiment and finetune, making it a promising tool for innovating AI solutions. Users can interact with the model through a user-friendly web UI for testing and customization, making Needle a notable advancement in the push for smaller, efficient AI systems.