94% on AIME with 1B Params (paradigma.inc)

Paradigma has unveiled Limite, its first model that boasts 1 billion parameters, designed specifically for tackling complex mathematical problems. This dense autoregressive transformer is trained on a carefully curated dataset of under 300 billion tokens, enabling it to manage sequences up to 131,000 tokens. Distinctively, Limite achieves a remarkable average score of 74.25% on the BeyondAIME math benchmark, outperforming larger models like MUSE-Glimmer-30B, with only a fraction of the computational resources. The model’s architecture is inspired by breakthroughs in pre-training methodologies, resulting in high sample efficiency and advanced problem-solving capabilities. Limite challenges the conventional view that models must adopt an assistant persona to be effective by emphasizing its mathematical reasoning over instruction following. This unique design allows Limite to proficiently solve intricate math problems while displaying a tendency to misinterpret non-mathematical inquiries. As Paradigma prepares for more extensive training runs, Limite sets a new benchmark in the AI/ML community for performance at lower parameter counts and paves the way for future models aimed at scientific and autonomous research applications. The release includes not only Limite but also evaluation data and custom plugins for optimized deployment.