Show HN: Self-host open-source LLMs on AWS with scale-to-zero (github.com)

A new tool called VeloxML has been introduced, allowing users to deploy open-source large language models (LLMs) directly onto their AWS or GCP accounts with a single command. This CLI and deployment engine, built on the robust SkyPilot orchestrator from UC Berkeley, simplifies the process, eliminating the need for Docker, Kubernetes, or deep cloud expertise. Users can launch a model service, provision instances, and set up a live API—all while ensuring that their data and model weights remain securely within their private network, thus avoiding compliance headaches associated with third-party services. The significance of VeloxML for the AI/ML community lies in its cost efficiency and ease of use. By using Spot instances, users can run AI models at minimal costs—around $0.07 per hour—without the financial overhead of idle GPUs. This tool promotes a serverless architecture tailored for small to medium-sized deployments, making advanced AI technologies accessible to a wider range of developers and startups. The ability to easily integrate private LLM inference APIs, batch processing, and real-time performance also enhances the capabilities for custom AI applications, positioning VeloxML as a game changer in the deployment of scalable AI services.