Train and run transformers directly on Apple's Neural Engine (github.com)

The recent launch of Espresso, a new framework for running transformer models on Apple's Neural Engine (ANE), marks a significant advancement in AI/ML performance on Apple Silicon. Espresso allows for inference that is 4.76 times faster than the conventional CoreML approach, achieving a decode time of just 1.08 milliseconds per token. It accomplishes this by compiling machine-independent language (MIL) programs directly to the ANE without the overhead of CoreML's intermediate steps, employing techniques like fused multi-layer kernels and zero-copy input/output. This paradigm shift enables developers to harness the full potential of Apple's hardware for advanced AI tasks. For the AI/ML community, Espresso opens up new possibilities in model training and inference directly on Apple devices, including full forward and backward passes with gradient accumulation. Built in Swift 6.2, Espresso offers zero dependencies, improving accessibility for developers transitioning to Apple’s framework. The private APIs used for this functionality prove that proprietary hardware can be effectively utilized for specialized AI tasks, potentially inspiring similar innovations in other ecosystems. As a result, Espresso not only enhances performance but also streamlines the development process, encouraging more robust AI applications on Apple platforms.