Squish – The fastest way to run local LLMs on Apple Silicon (squish.run)

Squish has launched an innovative local AI agent runtime designed specifically for Apple Silicon, allowing users to run any AI model quickly and completely offline. It boasts impressive performance, loading models in under a second—54 times faster than traditional methods—and provides a seamless experience without the need for cloud services or API keys. Users can install Squish via a simple Homebrew command, and with just one command, they can pull, optimize, and serve an AI model, thereby creating an instant chat interface in their browser—all while maintaining complete data privacy. This development is significant for the AI/ML community as it enables on-device inference, eliminating concerns about data privacy and costs associated with cloud services. Squish incorporates advanced features like INT4 compression to reduce model sizes and a unique caching architecture that improves speed and efficiency in processing multiple prompts. With built-in support for popular AI frameworks like LangChain and OpenAI's SDK, Squish aims to provide developers with a fast, reliable, and convenient solution for running AI models locally. The enhancements it offers, including optimized memory usage and guaranteed JSON syntax, position it as a potent alternative to existing local AI solutions.