LLM Classification Is Feature Engineering (minimallysufficient.com)

Recent discussions in the AI/ML community highlight the challenges and potential of using large language models (LLMs) as classifiers. While LLMs have shown a decent performance in classification tasks, issues such as poor calibration, difficulty interpreting results, and the inability to leverage structured data have raised concerns. The common practice of treating LLM outputs as hard labels limits their effectiveness, making it essential to rethink how we can utilize LLMs in a more structured, reliable manner. A promising approach is to consider LLM classification as a form of feature engineering, wherein the LLM’s outputs are integrated into traditional machine learning frameworks, like logistic regression. This method allows for better model calibration and the incorporation of various data types, enhancing both interpretability and performance. For instance, implementing logistic regression with LLM outputs can help refine decision thresholds and assist in understanding the contribution of LLM-derived features. By adopting this hybrid approach, researchers can significantly improve classifier performance, adapt to diverse datasets, and potentially unlock the broader capabilities of LLMs beyond mere text generation.