Statistics and ML Came to Have Two Different Kinds of Kernel Methods (bactra.org)

A recent discussion illuminated the differing interpretations of "kernel methods" in statistics and machine learning, two fields that often overlap yet employ distinct technical frameworks. The term "kernel" originates from integral operators in mathematical physics, but in statistics, it typically refers to convolution functions used in smoothing techniques, like kernel density estimation. In contrast, machine learning's kernel methods utilize two-argument functions associated with linear transformations, offering a powerful way to manage data by allowing implicit use of a vast set of basis functions through what's known as the "kernel trick." This distinction is significant for AI/ML practitioners as it emphasizes the conceptual and practical divergence in how kernels are applied across disciplines. The earlier approach, dating back to the 1950s, focused primarily on smoothing and estimation through convolution, while modern kernel methods in machine learning prioritize prediction and data modeling through complex, high-dimensional function spaces without needing explicit coefficient calculations. Understanding these nuances is crucial for researchers and practitioners aiming to apply the right technique in their AI and ML models effectively.