Show HN: An immutable local inference appliance with routing capability (github.com)

Sibyl has announced its new all-in-one inference server that transforms computers equipped with one or more GPUs into private AI servers. This pre-alpha tool can be effortlessly installed using a USB stick or network, and it operates via a web browser or terminal. Notably, Sibyl supports local models using llama.cpp to manage GGUF models, allowing them to utilize multiple GPUs simultaneously. The server boasts robust standard APIs compatible with OpenAI and Anthropic, ensuring that existing clients can integrate without modifications. The innovative model router feature automatically selects either a local model or a cloud provider based on user-defined rules, which can also ensure sensitive requests remain on the host. With added functionalities like a document search chat interface, extensive model management, and a secure host environment bolstered by Fedora's features, Sibyl stands out for its flexibility and security. Additionally, the platform is designed to accommodate various GPU vendors, providing unique host images tailored to different drivers. This development is significant for the AI/ML community as it presents a powerful, customizable solution for local AI inference while prioritizing security and usability.