Contrastive Language Model (CLM): An Ultra-Fast System One Model (twitter.com)

The Contrastive Language Model (CLM) has been introduced as an ultra-fast System One Model that utilizes a contrastive learning objective to effectively connect states and actions. With a pre-training foundation on internet-scale data, CLM-8B boasts inference speeds up to nine times faster than the Jev model while maintaining comparable performance in a variety of tasks, including computer use, gaming, and tool-calling applications. Impressively, it achieves state-of-the-art results on challenging coding benchmarks, such as DeepSWE and Terminal-Bench 2.1. The significance of CLM for the AI/ML community lies in its innovative training and serving infrastructure that disaggregates states and actions, enabling more efficient caching and reuse of embeddings. This approach substantially decreases inference latency, particularly in scenarios where the state changes continuously while the action set remains static. Additionally, the establishment of scaling laws for CLMs reveals a predictable decrease in test contrastive loss correlated to increases in training compute, model size, and dataset size, providing valuable insights for future model development and optimization.