I got tired of on-device LLMs crashing my apps, so I built a managed runtime (github.com)

Edge-Veda has introduced a groundbreaking on-device AI runtime for Flutter, designed to run language, vision, and speech models locally with less than 200ms latency, ensuring privacy and optimal performance. This system, which consists of approximately 22,700 lines of code and 50 C API functions, can operate without cloud dependencies, making it particularly appealing for mobile developers. By focusing on sustainability and long-term usability, Edge-Veda addresses common pitfalls of mobile AI implementations, such as thermal throttling, memory spikes, and session stability, which often hinder real-world applicability. Significantly, Edge-Veda offers developers structured observability, which facilitates real-time debugging and analysis, a crucial feature often missing in similar platforms. It adapts to thermal and memory conditions dynamically, ensuring persistent model operation across long sessions without crashes or model reloads. Key technical innovations include persistent inference workers, multi-turn chat management, and real-time audio transcription using GPU acceleration. With a built-in central scheduler and adaptive profiles for optimizing performance based on device metrics, Edge-Veda not only promises efficiency but also provides guarantees for battery and thermal management, making it a game-changing solution for local AI deployment in mobile environments.