Show HN: EdgeRunner – run GGUF models with Swift and Metal (github.com)

EdgeRunner, a new platform designed for running GGUF models on Apple devices, has been announced, enabling fast local inference of large language models directly on Mac and iPhone without requiring network access. Developed in Swift and optimized with Metal for Apple Silicon, EdgeRunner can load and execute GGUF models rapidly, boasting a median decoding speed of over 230 tokens per second on an Apple M3 Max. It supports various models, including those from the Llama family and offers features like streaming generation, memory-mapped model loading, and the ability to handle multiple quantization formats. This development is significant for the AI/ML community as it emphasizes on-device processing, ensuring data privacy by keeping conversations and interactions local. EdgeRunner facilitates the creation of private chatbots, code assistants, and embedded intelligence within mobile and desktop applications, reducing reliance on cloud services. The framework’s use of advanced Metal 4 capabilities further enhances performance, making it a compelling option for developers looking to integrate AI capabilities into their applications without incurring cloud computing costs. With ongoing support and improvements planned, EdgeRunner represents a promising step forward in the pursuit of efficient, private AI solutions.