Does Pretraining Teach Chemical Language Models Chemistry? (arxiv.org)

Recent research on chemical language models (CLMs) has raised critical questions about the effectiveness of pretraining in teaching these models chemistry. The study examined the performance of various CLMs across 64 datasets aimed at molecular property predictions, revealing that while pretraining does offer some benefits, the advantages do not scale, and the performance often plateaus below that of traditional gradient-boosting methods. Encouragingly, only supervised pretraining using standard physico-chemical descriptors consistently led to improved results. In many cases, models that were pretrained performed no better than those initialized randomly. This research is significant for the AI/ML community as it highlights limitations in the current approaches to model training for chemical applications. While CLMs are adept at understanding molecular structures, the study suggests they have not adequately internalized the chemistry beyond low-level statistical patterns. This finding emphasizes the need for integrating domain-specific knowledge into model training, indicating that future advancements in chemical informatics and molecular modeling may rely on refining pretraining methodologies to include deeper chemical insights rather than relying solely on large datasets.