Why feature selection methods rarely agree (I tested five side by side) (aayushig950.substack.com)

A recent hands-on comparison of five popular feature selection methods—Tree Importance, SHAP, Recursive Feature Elimination (RFE), Boruta, and Permutation Importance—applied to a realistic credit risk dataset reveals why these methods rarely agree and how their divergences carry valuable insights for ML practitioners. The study used a dataset mixing correlated, categorical, and skewed features under controlled preprocessing and splits to ensure fair comparison. While all methods converged on a core set of stable “all-weather” features, significant disagreements emerged around how they handle correlation, feature interactions, and contribution patterns. Tree-based importance favored features that create clear early splits but inflated importance for high-cardinality categoricals and arbitrarily chose favorites among correlated pairs. SHAP values provided a fairer attribution, highlighting “quiet influencers” whose consistent, subtle effects across samples tree methods missed. RFE prioritized features that contribute synergistically to model performance, sometimes rescuing variables overlooked by other techniques. Boruta was conservative, retaining only features clearly outperforming noise, while permutation importance explicitly flagged some features as harmful, illuminating noise or overfitting risks unseen by others. This analysis underscores that feature selection is not about finding a single truth but about integrating multiple “voices” to capture different predictive roles—split darlings, quiet influencers, synergy partners, or false friends. For the AI/ML community, it reinforces using diverse methods jointly to build robust, interpretable models and carefully handle correlated features to avoid misleading conclusions.