Building multilingual AI with a new open dataset (github.blog)

GitHub has launched the GitHub Multilingual Repositories Dataset, a new resource aimed at enhancing multilingual collaboration among developers by allowing them to discover public GitHub repositories with non-English content. This dataset, which offers metadata for over 40 million repositories, identifies language use in READMEs, issues, and pull requests, revealing noteworthy trends such as the prominence of Portuguese in READMEs and Korean in issues. By providing language classifications with confidence scores from multiple classifiers, it allows researchers and developers to tailor their exploration based on their specific needs. The significance of this dataset lies in its potential to address the underrepresentation of many European languages in AI systems, which often hampers the effectiveness of tools for diverse developer communities. This initiative aims to foster greater inclusivity by enabling the study of multilingual development practices and contributing to the creation of AI tools that better understand the unique language of software collaboration. Moreover, it opens avenues for improving evaluation methods and represents a broader commitment to supporting multilingual diversity in AI, encouraging developers and researchers to contribute to and build upon this resource for future applications.