What Least Squares Assumes About Your Data (sebmestre.blogspot.com)

Recent insights into least squares regression reveal its foundational assumptions and broader implications for modeling in the AI/ML community. Traditionally perceived as a straightforward means to minimize squared errors, least squares can actually be interpreted through the lens of maximum likelihood estimation (MLE), under the assumption that errors follow a Gaussian distribution. This alignment shows that least squares is not merely an arbitrary choice; it's rooted in a probabilistic framework that allows for deriving parameters (a, b) that best explain the observed data. This understanding invites practitioners to explore different error distributions—such as Laplace distribution—which result in alternative estimators that focus on minimizing absolute errors. The shift in perspective towards viewing models as assertions about data-generating processes encourages a more systematic approach to problem-solving. Instead of resorting to ad-hoc fixes when estimations falter, data scientists are urged to revisit and refine their models based on new insights. This iterative process fosters a deeper understanding of underlying assumptions and can reveal optimal solutions and contexts where certain methods may no longer hold, bridging the gap between theoretical modeling and practical application in real-world scenarios.