Curlie web directory download – 2.9M editor approved websites for your AI (curlie.org)

Curlie.org has announced the release of its comprehensive directory data for download, encompassing 2.9 million editor-approved websites. This significant resource offers a human-curated collection of high-quality, spam-free websites, organized into a well-structured, tree-like category hierarchy. The data is available under an Open Source license, allowing developers and AI researchers to utilize it in diverse applications, such as creating niche web directories or enhancing search engines. The collaboration with the Leibniz Supercomputing Centre ensures robust hosting capabilities, while OpenWebSearch.eu aims to integrate Curlie’s editorial data into its growing open web index. This initiative highlights the commitment to data transparency and accessibility within the AI/ML community, emphasizing the unique human ability to assess website trustworthiness compared to automated systems. The downloadable data is stored in a compact, textual format, making it easy for developers to explore its contents and implement ethical data practices. By making such a valuable dataset available, Curlie not only fosters innovation and collaboration within the tech space but also adheres to its open-source roots by encouraging contributions from the community.