PyTorch: A Reference Language (docs.pytorch.org)

PyTorch is increasingly viewed as both a reference and implementation language in deep learning, bridging the gap between clear API specifications and production-level efficiency. Traditionally seen merely as a user-friendly framework for training models, it has evolved into a vital tool that can also support optimized, high-performance operations through kernel domain-specific languages (DSLs). While these DSLs make it easier to achieve optimal performance for complex tasks like matrix operations, PyTorch still serves as a reliable reference implementation, enabling developers to verify the correctness of production code through numerical tests. The article proposes a novel approach for enhancing PyTorch's capabilities by utilizing coding agents and large language models (LLMs) to generate explicit forward-backward versions of code for optimization, while maintaining the original PyTorch code as a reference. This method mitigates concerns regarding the divergence between the reference and production implementations, suggesting the implementation of verifiers to ensure equivalence. This innovative perspective not only addresses the challenges of traditional autograd techniques but also reinforces PyTorch's critical role in the evolution of AI/ML practices as the field scales. The ongoing dialogue raises important questions about achieving the balance between performance and usability in deep learning frameworks.