Intuit's Numaflow Abstracts Away Infrastructure for ML Engineers (thenewstack.io)

Intuit unveiled Numaflow, an open-source, Kubernetes-based stream processing engine (from the team that built Argo CD) designed to hide infrastructure complexity so ML and data engineers can build real-time pipelines without deep Kubernetes or Java/Scala expertise. Announced at Kubecrash 2025, Numaflow provides a UI and declarative YAML pipeline definitions, connects to Kafka, Pulsar and SQS, and runs user logic as UDFs (Python/Java). Pipelines are built from vertices (compute units) that read abstracted sources and write to sinks, allowing each vertex to scale independently and automatically based on incoming load instead of hand-tuning pods. For the AI/ML community this matters because stream processing is central to feature engineering, online inferencing, training-data refreshes, real-time recommendations and fraud detection. Numaflow’s serverless abstraction eliminates repeated boilerplate (connectors, queue handling, scaling concerns) so engineers can focus on payload and model logic. Technical highlights: per-vertex autoscaling tied to event backlog (removing manual HPA tuning), UDF-based inference workflows, a GUI that visualizes running pods and pipeline topology, and production-proven use cases (image recognition demo and a year-old anomaly-detection pipeline). Combined with Argo, Numaflow promises a compact, Kubernetes-native stack for building scalable, real-time ML systems.