Scaling Indic Pretraining: 2.8T Tokens and an Open Data Generation Pipeline (huggingface.co)

AI4Bharat, in collaboration with NVIDIA and IIT Madras, has announced the upcoming release of the Nemotron-pretraining-Indic-v1, a groundbreaking corpus comprising approximately 2.8 trillion tokens across 22 Indian languages. This extensive dataset, designed to aid developers in building AI models for Indian languages, also includes a GPU-accelerated data generation pipeline. By addressing the significant data deficit in Indian language technology, this initiative targets the needs of nearly 1.2 billion speakers, enabling enhanced training for models to explain technical concepts and provide answers in these languages. The thorough nature of the corpus, which incorporates checks for translation quality and includes both native-script and romanized formats, ensures that generated synthetic text remains grounded in accurate, curated sources. This initiative is significant for the AI/ML community as it significantly expands the resources available for Indic language research, previously hindered by a lack of data. The open data generation pipeline allows researchers and developers to not only utilize the corpus but also adapt the data generation processes to enhance their own datasets. Key technical features include advanced translation quality assessments using a rubric inspired by Multidimensional Quality Metrics, ensuring high fidelity in the generated content. The corpus is expected to improve training outcomes as evidenced by initial results from the Nemotron 3.5 Super VL model, which demonstrated notable performance gains across various Indic language benchmarks. Overall, this initiative is a substantial step toward democratizing AI capabilities in India’s diverse linguistic landscape.