Like Ollama, but for your own cloud [Apache 2.0] (github.com)

The recent announcement of the SIE open-source inference engine marks a significant advancement for the AI and machine learning community by providing a unified API for serving embeddings, reranking, and entity extraction across more than 85 pre-configured models. This new system replaces the need for disparate model servers, enabling seamless integration from local environments to Kubernetes clusters. The engine supports diverse architectures, including dense, sparse, multi-vector, and vision models, streamlining the embedding and extraction processes for developers. SIE's architecture includes essential components such as a load-balancing gateway, KEDA for autoscaling, and customizable dashboards via Grafana, all packaged to simplify deployment on platforms like GKE and EKS. Its compatibility with major frameworks such as LangChain, Haystack, and Weaviate enhances its utility in real-world applications. With features like on-demand model loading and a drop-in migration path for users of OpenAI’s API, SIE aims to provide a versatile and scalable solution for businesses leveraging AI. This deployment-ready stack and its comprehensive model offerings position SIE as a critical tool for accelerating AI workflows.