BigBang-Proton, Next-Word-Prediction Is Scientific Multitask Learner (arxiv.org)

BigBang-Proton is a new auto-regressive, sequence-based model proposed as a "scientific multitask learner" trained by next-word prediction on a mixture of cross-scale, cross-structure, cross-discipline scientific datasets plus general text. The paper highlights three architectural innovations: Theory-Experiment Learning, which explicitly aligns large-scale numerical experimental data with theoretical text corpora; Binary Patch Encoding, a replacement for BPE tokenization designed for mixed numeric/text inputs; and Monte Carlo Attention, a novel alternative to standard transformer attention. The model is pretrained on real-world scientific problems and then fine-tuned for downstream tasks, preserving a unified next-token objective across domains. Reported results are striking: exact 100% accuracy on up-to-50-digit addition, parity with specialized models on particle-physics jet tagging, matching MAE for inter-atomic potential simulations, comparable performance to spatiotemporal models for water-quality forecasting, and benchmark-beating results in genome modeling. If reproducible, these outcomes suggest language-guided modeling can match or exceed task-specific scientific models while offering a single multitask foundation for heterogeneous scientific problems. The authors further propose scaling pretraining to an extreme “universe scale” to build a material-world foundational model; the claims are promising but will require independent replication and peer review.