I Built a System One Model on a RTX 3070 (twitter.com)

A developer has created an innovative AI model called System One, specifically designed for short English text processing and trained on an RTX 3070 GPU. This system employs a unique approach where it accepts a state and a set of typed questions, utilizing three key primitives: choice, score, and noul. Unlike traditional models that generate text, System One produces structured responses directly from model scores, streamlining the output process. The foundation of the model is a pretrained ModernBERT-base, enhanced with LoRA adapters and a special decision head, allowing it to evaluate multiple questions efficiently within a constrained token limit. Significantly, the training of System One occurs in two stages. Initially, it uses human-labeled data from diverse datasets like BANKING77, SST-5, and SMS Spam, employing a combination of cross-entropy and ranked probability scores to enhance prediction accuracy. The second stage involves teacher distillation, where the model learns by mimicking a larger pretrained model's outputs. While the model achieved impressive scores — 98.9% on SMS spam detection and 88.5% on BANKING77's intent classification — it highlights the importance of task-specific human supervision for better training results. This project suggests a promising avenue for automating processes by selectively handling confident predictions, which could improve efficiency in real-world applications. The code and weights are available on GitHub and Hugging Face, respectively.