Taming the ML Firehose: Scaling Feature Consistency (www.uber.com)

Uber has announced a significant advancement in its machine learning processes with the introduction of a feature logging framework designed to enhance feature consistency across online and offline training pipelines. The initiative addresses critical issues where discrepancies in feature formats can lead to performance regressions in machine learning models. By ensuring that the same features are utilized at both training and inference stages, Uber aims to strengthen model performance while expediting the identification and resolution of problems. The framework tackles challenges related to data volume and bandwidth constraints, enabling the logging of essential features without overwhelming storage systems. Key technical strategies include implementing a feature allow list to reduce payload sizes, utilizing feature name aliasing to minimize data transmission costs, and employing selective logging to capture only impactful predictions for training. Moreover, by optimizing state management for distributed systems and enhancing observability metrics, Uber has achieved a 0% mismatch in key feature logging and improved data freshness from days to hours. These enhancements not only decrease infrastructure costs but also contribute to a more reliable and efficient machine learning model training environment, ultimately facilitating faster iterations and better performance outcomes.