Show HN: Orion – Native Training LLMs on the Apple Neural Engine Without CoreML (github.com)

Orion, a newly announced AI runtime, enables the training and execution of small language models (LLMs) directly on Apple Silicon's Neural Engine (ANE) without relying on Apple's CoreML framework. This innovative approach allows developers to run AI applications entirely offline on devices such as iPhones, iPads, and Macs, benefiting the burgeoning AI/ML community by providing greater freedom and efficiency in deploying LLMs while circumventing the limitations imposed by CoreML's compilation and operational restrictions. Significantly, Orion demonstrates the ability to perform both training and inference on the ANE, utilizing its dedicated machine learning capabilities that many applications fail to capitalize on. It achieves up to 170+ tokens per second with the M4 Max chip, presenting a compelling performance benchmark for local AI applications. Key technical details include Orion’s unique compilation and memory management capabilities, which enable direct access to ANE's hardware while supporting advanced features like program caching and budget-aware compilation, making it a versatile toolkit for developers. With the potential for applications in privacy-sensitive environments and edge deployment, Orion paves the way for more efficient, localized AI solutions.