Show HN: Smile-Serve – Inference Server for ML, ONNX, and LLM (github.com)

SMILE Serve has been launched as a production-ready inference server designed for machine learning models, supporting Classic ML, ONNX, and LLM functionalities on the JVM. Built on Quarkus, it allows users to serve serialized SMILE models (.sml), ONNX models from various frameworks, and provides a chat interface utilizing the Llama 3 model. The server comes with a straightforward setup through Docker, enabling easy deployment of models for inference by simply mounting local model directories. This development is significant for the AI/ML community as it bridges the gap between different model formats and offers an efficient, unified API for model inference. The ability to handle multi-type models (Classic ML, ONNX) and maintain a chat interface in one server instance streamlines workflows for developers. Key technical features include live-reload development, automatic model discovery at startup, and efficient handling of model metadata and input/output management, which enhances usability and accelerates deployment for real-time AI applications.