I benchmarked every model that fits on an iPhone (digital-foundry-eight.vercel.app)

A recent benchmarking study evaluated various AI models' performance on iPhone, directly challenging the common debate among iOS teams: whether to use Apple's on-device system model, integrate their own models via MLX, or rely on a cloud API. The study revealed that Apple's Foundation Models system model excelled in throughput, achieving approximately 149 tokens per second (tok/s) while maintaining a low peak app memory of just 12MB. In contrast, models like Qwen3 4B and Llama 3.2 3B suffered under thermal constraints at higher parameters, causing significant drops in performance. The implications of these findings are noteworthy for the AI/ML community. A key takeaway is that the system model's real advantage lies not just in speed but in memory efficiency, allowing developers to run powerful applications without exceeding memory limits. Furthermore, the study highlighted that 4-bit quantization could be utilized without noticeable degradation in quality, effectively halving memory load. As the demand for mobile AI applications rises, understanding these performance metrics could guide developers in making informed choices about model deployment on iOS devices.