A Trick for Backpropagation of Linear Transformations (tripplyons.com)

A recent explanation highlights a simple yet powerful trick for backpropagating through any linear transformation expressed via einsum (Einstein summation) notation, a versatile tool that can represent operations like sums, matrix products, dot products, and Hadamard products. The core insight is that to compute gradients during backpropagation, you can reuse the forward einsum expression by swapping the input and output indices appropriately. For example, the gradient of the loss with respect to an input matrix \(A\) in a matrix multiplication \(C = A @ B\) can be efficiently computed by swapping the indices in the einsum expression, effectively multiplying the upstream gradient by the transpose of \(B\). This technique greatly simplifies deriving backpropagation formulas for complex linear operations, providing a clean, uniform approach that avoids tedious manual gradient computations. By aligning the output indices with the input tensor’s shape, the method ensures shape correctness and interprets the backward pass as another einsum operation, often analogous to a matrix multiplication with a transposed term. The approach is validated using JAX’s automatic differentiation, confirming that the manually derived gradients match autograd results exactly. For the AI/ML community, this trick streamlines gradient computation in custom models and can be particularly useful in frameworks where explicit einsum operations are involved. It enhances both clarity and efficiency in backpropagation of linear layers, encouraging more expressive use of einsum notation in differentiable programming and model design.