Show HN: I trained a 1B LLM from scratch for $315 and open-sourced weights+data (huggingface.co)

AIIT-THRESHOLD has announced the release of Tessera 1B, a new ~1 billion-parameter language model trained entirely from scratch for just $315. This model utilizes a hand-curated corpus of 24.5 billion tokens, offering open-source weights and data to the AI/ML community. While Tessera 1B demonstrates fluent English and some Japanese capabilities, it is not designed to serve as a ready-to-use chat assistant or a reasoning model. Instead, it functions as a robust foundational model that can be fine-tuned for specific applications, enabling researchers and developers to leverage its clean architecture for tailored solutions. The significance of this release lies in its affordability and accessibility, facilitating experimentation and innovation in AI model development. Employing a custom decoder-only transformer architecture dubbed "ProtoGPT," Tessera 1B features 32 layers and supports a context length of 4096 tokens. Despite its limitations—such as a final evaluation loss of ~3.20 nats and inconsistent performance on factual recall—the model represents an honest approach to AI training, devoid of synthetic reasoning traces or conversational data. This transparency allows users to understand its capabilities and seize the opportunity to create specialized models by fine-tuning it for their unique needs.