Why not to use the Gaussian kernel (arxiv.org)

A new analysis raises critical concerns about the use of the Gaussian kernel, also known as the squared exponential or radial basis function kernel, in machine learning applications like Gaussian process regression. The study argues that this popular kernel should not be the default choice due to its inherent brittleness. Key findings indicate that the Gaussian kernel leads to an unrealistically small conditional variance, resulting in significant overconfidence in predictive uncertainty assessments. Additionally, this small variance is associated with numerical instability, necessitating modifications such as nugget terms that alter the underlying model. The implications of this research are significant for the AI/ML community, as it challenges long-standing practices around kernel selection. The authors urge practitioners to reconsider the use of analytic kernels due to their tendency to produce excessively smooth predictions, which can undermine model reliability. By promoting awareness of these pitfalls, the study aims to foster more robust modeling techniques and enhance the overall quality of predictions in machine learning tasks.