Benchmarking On-Device Machine Learning on Apple Silicon with MLX (arxiv.org)

Researchers introduced MLX, a framework optimized for on-device ML for Apple Silicon, and a companion toolkit, MLX-transformers, to benchmark transformer inference on MacBook hardware. The study measures inference latency for BERT, RoBERTa and XLM‑RoBERTa implementations in MLX against PyTorch counterparts and an NVIDIA CUDA GPU baseline, using models with identical parameter counts and checkpoints. MLX‑transformers includes multiple transformer implementations, automates checkpoint handling (converting PyTorch checkpoints to MLX format) and also enables running Hugging Face models more seamlessly on Apple Silicon without the usual porting friction. The work is significant because it quantifies how Apple’s chip architecture can support efficient, low-latency transformer inference on consumer devices—lowering barriers to offline, private, and edge deployment for research and prototyping. Key technical takeaways: benchmarking focuses on latency (inference) across two MacBook Apple Silicon devices, compares implementation-level differences with PyTorch, and demonstrates MLX’s viability for running real-world transformer workloads locally. Results highlight MLX’s potential to make on-device ML more accessible in Apple’s ecosystem and lay groundwork for extending evaluation to other model families and modalities.