Discord Creates Insights from Trillions of Data Points (2021) (discord.com)

Discord’s Data Platform team has developed a robust system—internally known as Derived—that leverages over 15 trillion records and petabytes of raw data to enhance data analysis and support product decisions. Initially relying on manual computations, Discord has scaled its operations by processing data with 30,000 vCPUs in the cloud, transforming raw data into a structured data warehouse using Google BigQuery. The Derived system utilizes a Directed Acyclic Graph (DAG) of precomputed tables to streamline data transformation and access for various stakeholders, including engineering and data science teams, significantly improving efficiency and decision-making capabilities. The significance of this initiative for the AI/ML community lies in its innovative approach to data management and automation within a fast-growing tech environment. By simplifying the creation and modification of derived tables through SQL and YAML, Discord has enabled teams to autonomously handle data operations while ensuring adherence to privacy policies and governance. This system not only enhances the responsiveness of data insights for product features but also focuses on automation and error-reduction in table management, allowing for parallel computations that maintain consistency and timeliness of derived insights. This development exemplifies how tech companies can address complex data challenges while fostering a culture of self-service and agility in data handling.