Learning better decision trees – LLMs as Heuristics for Program Synthesis (mchav.github.io)

A recent experiment has explored the use of large language models (LLMs) as heuristics to enhance decision tree learning through improved feature engineering. By treating this process as a program synthesis problem, the researcher generated candidate features from raw data and employed an LLM to filter out nonsensical arithmetic expressions. The LLM's role was to assess whether the generated features represented coherent, interpretable quantities—promoting meaningful feature combinations while tuning out those grounded purely in statistical correlation. This approach is significant for the AI/ML community as it illustrates a novel application of LLMs to automate and improve the interpretability of machine learning models without loss of rigor. The study utilized Kaggle's Titanic dataset to demonstrate that expressions deemed meaningful by the LLM produced decision trees with greater human-readable structures and slightly enhanced predictive accuracy. By scoring candidate features based on their practicality and coherence, this method could help streamline feature engineering, a traditionally manual and subjective task, ultimately leading to more robust and interpretable models in various applications.