A polynomial autoencoder beats PCA on transformer embeddings (ivanpleshkov.dev)

A recent study has demonstrated that a polynomial autoencoder (poly-AE) significantly outperforms traditional Principal Component Analysis (PCA) on transformer embeddings, particularly in capturing the complex, nonlinear structures inherent in these embeddings. While PCA is effective for linear projections, it fails to account for variance residing in the nonlinear tail of the data distribution. The proposed method uses PCA to encode the data and employs a quadratic decoder, implemented as a closed-form Ridge regression, to improve the decoding process without engaging in extensive hyperparameter tuning or iterative optimization. This innovation leads to a notable enhancement in retrieval quality, with the polynomial decoder yielding improvements in the Normalized Discounted Cumulative Gain (NDCG) metric across various models, particularly at lower dimensions. The findings hold significant implications for the AI/ML community, as they reveal how integrating techniques from dynamical systems can enhance retrieval methods for neural embeddings. In experiments using standard datasets, the poly-AE achieved memory compression of up to eight times while maintaining a negligible drop in performance compared to raw embeddings, showcasing its efficiency and effectiveness. This closed-form approach not only streamlines the compression process but also suggests that leveraging quadratic formulations can be a valuable strategy for optimizing model inference, particularly in resource-constrained environments.