Standardized and In-Depth Benchmarking of Post-Moore Dataflow AI Accelerators (arxiv.org)

A groundbreaking framework called DABench-LLM has been introduced for standardized and in-depth benchmarking of Post-Moore dataflow AI accelerators, specifically aimed at large language models (LLMs). As the rapid growth of LLMs surpasses the capabilities of older CPU and GPU architectures, this framework represents a significant advancement by providing a comprehensive method for evaluating LLM workloads across various dataflow-based accelerators. DABench-LLM combines intra-chip performance profiling with inter-chip scalability analysis, addressing key metrics like resource allocation, load balance, and resource efficiency. This initiative is crucial for the AI/ML community as it fills the existing gap in performance analyses for dataflow AI accelerators, which have not been extensively benchmarked until now. By validating DABench-LLM on popular platforms such as Cerebras WSE-2, SambaNova RDU, and Graphcore IPU, researchers can identify performance bottlenecks and receive targeted optimization strategies. This innovation not only accelerates the process of gaining insights into hardware capabilities but also enhances the potential for optimizing LLM training across diverse AI hardware setups, paving the way for future advancements in AI model development.