Running Laguna S 2.1 locally on Apple Silicon: 52 tok/s with 38.5 GB peak memory (github.com)

A new local benchmarking repository for the Laguna S 2.1 machine learning (ML) model has been introduced, specifically optimized for Apple Silicon, demonstrating impressive performance metrics. Conducted on a MacBook Pro equipped with an M5 Max chip, tests showed that the smallest quantization version, oQ2e, achieved a throughput of 52 tokens per second while using a peak memory of 38.5 GB across various tasks. This quantization outperformed other tested frameworks in both speed and task completion, marking a significant advancement for running ML models on Apple’s ARM architecture. The benchmark results have broader implications for the AI/ML community by providing developers and researchers with a robust framework to compare different quantization strategies locally. The harness also gives insights into memory usage and task performance across several scenarios, including generation and agentic tasks, crucial for refining model efficiency. The standardization of benchmarking processes simplifies the evaluation of future models on Apple Silicon, helping drive innovation and adoption of this hardware in AI applications.