Private and OpenAI-Comp. Inference Engine for Nvidia Linux and Apple Silicon (github.com)

The Tara Vision Inference Engine was announced as a local, private, and OpenAI-compatible solution for running AI models like Gemma, GLM, and Qwen on both NVIDIA Linux machines and Apple Silicon Macs. Built on llama.cpp and designed for taracode, it ensures a privacy-first experience with no telemetry and requires no user accounts for access. Key features include a pinned runtime for robust version control of llama.cpp builds, a curated model catalog optimized for specific hardware, and a straightforward installation process through a single command. This development is significant for the AI/ML community as it allows users to leverage powerful AI models locally while maintaining privacy and control over their data. The system’s architecture uses a combination of the llama-server API for compatibility with OpenAI and Anthropic models, along with detailed measurement metrics for performance monitoring. The engine is fully open-source, MIT-licensed, and installs easily, making state-of-the-art AI accessible to a broader audience without the security concerns associated with cloud-based services.