Can a Linear Program Do Weight-Space Programming? (nathan.rs)

MIT's CSAIL recently unveiled a groundbreaking technique called "weight-space programming," exemplified by their model, GPT-6 Astra. The model demonstrated an ability to accurately identify its training games solely from its raw weights and, given just a game’s rules, generate a network that outperformed traditional models trained with Proximal Policy Optimization (PPO) after extensive iterations. Notably, Astra improved the math accuracy of another model, Qwen3.5-0.8B, by fine-tuning merely 0.016% of its weights, significantly boosting its performance on math tasks from 7.9% to 48.1%. This presents a significant advancement in how models can be modified without traditional training approaches, emphasizing reasoning over gradients. Astra’s updates mainly targeted loops in the model's reasoning pathways, which hinder its output quality. The techniques involved solving linear programming (LP) problems to determine which weight adjustments would best mitigate repetitive looping. While Astra's original exploration took hours of computational time, the researchers found that LP could streamline this process by utilizing fewer weight changes to achieve comparable improvements in accuracy. This methodology not only simplifies the weight adjustment process but also opens the door to broader applications by potentially enabling similar techniques on other models, establishing a new paradigm for enhancing AI performance.