Any Engine, Any Topology, Any Infrastructure: How We Designed Modelplane (modelplane.ai)

Modelplane, a new fleet-level API for inference clusters, has been launched to address the limitations of existing cluster-level tooling. While current tools handle single clusters efficiently, as organizations scale to tens or hundreds of clusters across various infrastructures, they face challenges such as fragmented placement, isolated capacities, and redundant development efforts. Modelplane allows teams to describe deployments once, utilizing any engine and topology on available hardware, promoting a more unified and efficient approach to inference. Built on Crossplane, Modelplane separates the roles of platform teams and developers through a structured API. It abstracts details of the underlying engines and focuses on the shape and structure of model deployments instead. This design choice allows for compatibility with rapidly evolving engines without needing frequent API updates. Key features include the ability for platform teams to declare the fleet's hardware capabilities while developers specify workload requirements, enabling seamless scheduling and efficient resource utilization across diverse infrastructures. Currently at version 0.1, Modelplane encourages community engagement for further refinements and enhancements, making it a significant step toward scalable, flexible inference management in AI/ML environments.