Show HN: I trained a language model that thinks the capital of Japan is Paris (hamiltonianresearch.xyz)

In an intriguing project shared on Show HN, a 13-year-old developer has trained a language model named DIMBA II that humorously believes the capital of Japan is Paris. This work, although lighthearted in its inception, sheds light on significant advancements in model architecture. DIMBA II innovatively combines the efficiency of Mamba-2 and the parallel generation capabilities of diffusion language models, marking a departure from traditional transformer architectures to address the growing challenges of scaling AI efficiency with increasing context lengths. The significance of DIMBA II lies in its exploration of masked diffusion, a technique that has not typically been employed outside transformer frameworks. Key improvements over its predecessor include a focus on masked diffusion to directly learn and generate text, along with structural optimizations such as a critic head for output evaluation. Despite encountering hurdles during training—most notably issues with distillation and latent-space representation—the model demonstrates an intriguing mix of potential and limitation, achieving a 20% factual accuracy in quality assessments against established models. This experiment not only showcases innovative attempts to wring more value from smaller models but also emphasizes the importance of architecture design in overcoming current performance obstacles in AI language processing.