Optimizing Our ML Feature Store: Cutting Compute Costs (kayhan.dev)

A recent update revealed significant enhancements to an ML feature store's compute layer, focusing on reducing processing costs and improving efficiency. The team replaced AWS Fargate with Karpenter for batch compute, achieving pod startup times reduced from minutes to seconds. This change, combined with multi-architecture Docker images, effectively doubled the available spot instance pool and cut compute costs by approximately 55-60%. By adopting a 60/40 split between spot and on-demand EC2 instances, they minimized the risk of job interruptions while maximizing resource use. Additionally, the team advanced memory allocation techniques by implementing quantile regression models to predict resource needs more accurately, addressing underutilization issues that once left up to 50% of memory idle. By using defined input schemas, they attained consistent job behavior, leading to a 40% reduction in resource waste. This dual approach not only streamlined compute costs but also refined overall job performance, positioning the feature store as a robust and efficient solution for machine learning tasks, with the potential for broader implications in feature engineering workflows across the AI/ML community.