Quaternion small language-model comparison (github.com)

A recent experiment compared quaternion, real, and complex Transformative models in a compact Karpathy-style decoder setup. While maintaining consistent data, batching, and training parameters, the study monitored how different projection types affected model performance. Initial findings indicated that quaternion projections outperformed real projections in early training, with lower validation loss observed up to 2 million training tokens. However, from 10 million tokens onward, real projections demonstrated superior performance, suggesting a complex interplay between model architecture and data volume. This research is significant for the AI/ML community as it explores the potential of quaternion representations in enhancing model efficiency, particularly in the early stages of training. The results could prompt further investigation into how various mathematical frameworks impact learning dynamics. As AI continues to push the boundaries of model capabilities, understanding these nuances will be crucial for developing more efficient and effective AI systems. The careful methodological approach, including rigorous validation and detailed logging of experiments, strengthens the study's credibility and provides a framework for future research in this area.