On The Architectural Complexity of Neural Networks (arxiv.org)

A new paper introduces a unified theoretical framework for analyzing and constructing deep neural networks (DNNs), addressing a critical gap in existing methods by explicitly modeling tensor operations, which are often abstracted in prior approaches. This framework allows researchers to not only assess the evolution of architectural complexity in deep learning but also to automatically generate new architectures using innovative tensor operations. The study highlights a correlation between major advancements in DNN architectures over the past four decades and the complexity inherent in their designs. Significantly, the authors have compiled a dataset of over 3,000 unexplored, higher-complexity architectures, which they will publicly release. This resource could expedite research and experimentation in the AI/ML community, unlocking the potential for novel neural network designs. By providing a systematic approach to both analysis and construction, this work fosters deeper insights into DNN evolution and encourages the exploration of intricate architectures that have been overlooked, paving the way for future breakthroughs in AI.