Temporal Language Models (www.calcifercomputing.com)

A new approach to language modeling called Temporal Language Model 1 (TLM-1) has been introduced, which addresses a significant gap in traditional Large Language Models (LLMs) by incorporating the inherent temporal structure of language. Unlike previous models that do not account for the order in which documents were created, TLM-1 combines content prediction with document date classification, allowing it to analyze linguistic trends over time. Trained on a general-purpose corpus of American English spanning from 1990 to 2019, TLM-1 uniquely enables researchers to explore how language has evolved across different genres, thanks to its innovative joint learning methodology. The significance of TLM-1 for the AI/ML community lies in its enhanced capacity to retrieve and recover temporally sensitive relationships in language, which traditional models might overlook due to their atemporal nature. By employing a Bayesian framework to disentangle sources of temporal bias, TLM-1 opens new avenues for analyzing historical linguistics more robustly. Its ability to learn time embeddings that represent temporal dynamics effectively predicts document dating and elucidates the progression of word meanings and usages, thus providing a versatile tool for temporal language tasks without the narrow constraints faced by earlier models like TempoBERT.