The next 700 ML model-serving platforms (eff-kay.github.io)

Researchers laid out a compact grammar for ML serving — ModelArtifact → Container → Endpoint → Version with operations package, deploy, scale and route — and used it to map how major platforms implement model serving. AWS SageMaker most closely follows the formal model with explicit container building, endpoint management and traffic weights; Azure ML is workspace- and registry-centric with strong environment/conda management and managed online endpoints; Google Vertex AI emphasizes cloud container integration and GPU-accelerated machine types; ServerlessML removes containers/endpoints entirely, treating models as auto-scaling functions with cold-start policies; and StatefulML makes caching and warm state first-class, exposing predictive caches and incremental update strategies. Example details include container images and runtime envs (ECR/GCR/MCR), scaling_config fields (instance_type, min/max replicas, accelerator_count), traffic split semantics, and function-level scaling rules or cache eviction strategies. This comparison matters because it exposes concrete trade-offs — control vs. simplicity, explicit infrastructure vs. serverless abstractions, stateless throughput vs. stateful performance optimizations — and shows that many differences are platform design choices rather than inherent requirements. For ML teams this implies portability, observability, and cost models will vary widely; the paper argues for more consistent, composable abstractions (dynamic resource allocation, shared model loading, predictive monitoring, cross-platform routing) to make deployments reproducible, efficient and easier to reason about across clouds.