talkie-coder: From 1930 to SWE-bench (github.com)

Researchers have successfully fine-tuned the "talkie-1930" vintage language model (LLM), initially pre-trained on data only up to 1931, to address coding challenges in the SWE-bench benchmark. By using just 250 training examples, the model achieved its first fix, notably improving its performance from a 4% pass rate to 4.5% pass@1 on SWE-bench-Verified after scaling training to 75,000 trajectories, equivalent to 1 billion tokens. A sibling model trained on web data outperformed this with a pass@1 rate of 5.5%, demonstrating minimal loss of capability despite the reduced dataset, emphasizing the model's potential for effective code-based reasoning without extensive internet training. This development is significant for the AI/ML community as it showcases the feasibility of fine-tuning historical machine learning models to solve contemporary problems, like software engineering assessments. The detailed approach included scaling training methods, utilizing advanced context mechanisms, and employing transfer learning techniques that could guide future research in optimizing older models for practical applications. The project shares extensive resources, including scripts and evaluation pipelines, for replicating and examining the model's performance, potentially paving the way for further innovations in language models and their adaptations.