Soofi: Complete training code for an open-source foundation model (github.com)

The Soofi project has announced the release of its complete training code for an open-source foundation model, specifically the Soofi S, a 30 billion parameter model built on Nvidia's Nemotron 3 Nano architecture. The repository includes extensive documentation and scripts covering various aspects of the model's pretraining phase, such as corpus construction, filtering, tokenization, and assembly. Key components of the training process are organized into distinct directories, facilitating users' access to preprocessing and midtraining stages. This announcement is significant for the AI/ML community as it promotes transparency and collaboration in model development. By providing the full codebase and upcoming model weights, Soofi enables researchers and developers to replicate, adapt, and build upon the groundbreaking techniques used in the training process. This could accelerate advancements in large language models and foster a more inclusive environment for innovation in AI research.