Sharded repodata in conda (beta): an order of magnitude faster fetching (conda.org)

Conda has introduced a beta feature called sharded repodata, which significantly improves the performance of package metadata fetching by reducing the data size required for downloads. This new format optimizes how conda clients access package information, allowing for faster environment solves—up to ten times quicker when fetching metadata. By splitting repodata into smaller "shards" for each package, users now only download what’s necessary, drastically cutting down both download times and memory usage. The sharded repodata implementation, guided by CEP 16 authored by Bas Zalmstra, has already been adopted by conda-forge, enhancing the efficiency of package management in the conda ecosystem. This development is particularly beneficial for users and maintainers as it decreases the frequency of cache invalidations and overall builds times, leading to a smoother user experience. With a 35-fold reduction in data downloaded and a substantial decrease in memory usage, this feature promises to streamline package installations and environment setups across the AI/ML community using conda. Users can easily enable this feature by updating their conda environments and opting in to sharded repodata.