96% Correct Next Token Prediction, with No DNN, No Training, Autodistilled Model (mltechniques.com)

Vincent Granville has introduced a groundbreaking model for next-token prediction that achieves 96% accuracy on unseen data without relying on deep neural networks (DNNs) or traditional training methods. This innovative approach utilizes auto-distillation to efficiently identify and eliminate non-contributory weights, resulting in a lightweight model capable of generating highly accurate predictions for specialized corporate corpuses. Compared to standard transformer models, which typically perform at 30-55% accuracy, Granville's methodology demonstrates superior performance while significantly reducing computational costs. The model's architecture is inspired by RBF (Radial Basis Function) networks, integrating concepts familiar in various research fields. This not only enhances explainability but also allows for a controlled degree of randomness in responses, which can be tailored as needed. With promising results in the NVIDIA case study, the model's potential applications extend beyond token prediction to areas such as image classification and numerical data analysis. Granville plans to further develop this methodology using larger corpuses, expanding its utility and impact within the AI/ML community.