DiffusionBench: Towards Holistic Evaluation of Generative Diffusion Transformers (github.com)

A new benchmarking framework called DiffusionBench has been announced to facilitate the evaluation of generative diffusion transformers across various tasks, including image generation and text-to-image transformations. This unified codebase allows researchers to train and evaluate models on multiple datasets through a single interface, addressing the limitations of traditional evaluation methods that often rely solely on ImageNet metrics. The initiative encourages contributions from the community to expand the benchmark with new evaluation axes and metrics. DiffusionBench's significance lies in its comprehensive approach to assessing generative models, incorporating advanced evaluation methods such as FID (Fréchet Inception Distance) and IS (Inception Score), tailored for different types of generative tasks. The framework includes multiple training stages for different model architectures—like VAE, RAE, and pixel-based models—while supporting detailed reproducibility across various settings. This development could significantly streamline the evaluation process in the AI/ML community, fostering more rigorous comparisons of diffusion models and accelerating advancements in generative AI research.