We run Gemini at scale across billions of posts (www.modash.io)

Modash has successfully integrated large language models (LLMs) into its expansive creator-discovery dataset, which processes billions of social media posts daily. The shift to LLMs addresses the limitations of traditional data extraction methods, which struggled with context and accuracy in multilingual, multimodal content. The transition has significantly enhanced the quality of the data delivered to clients, minimizing false positives in identifying sponsored content, a major concern for customer satisfaction. To handle the high volume of inference required by LLMs, Modash employs a sophisticated multi-cloud architecture utilizing AWS and GCP services. This system optimizes resource use by distributing load across multiple regions and employing batch processing to manage costs effectively. Key techniques include encapsulating data requests into large JSONL files to reduce operational complexity and using strategic prompt engineering to maximize performance while minimizing token usage. By refining prompt formulations and enhancing result schemas, Modash has improved efficiency, allowing for accurate data extraction without incurring exorbitant operational expenses, thereby setting an example for other organizations in the AI/ML community aiming to scale LLM capabilities economically.