Reproducibility Test-Time Training on Nearest Neighbors for LLMs (arxiv.org)

A recent reproducibility study explored the efficacy of Test-Time Training (TTT) on Nearest Neighbors for Large Language Models (LLMs) and confirmed significant performance enhancements. Researchers reproduced the methods presented in Hardt and Sun (2024), utilizing pretrained RoBERTa embeddings to retrieve 20 neighbors per test input and performing gradient updates on varied models, including GPT-2 and GPT-Neo. Their experiments demonstrated that TTT effectively reduces perplexity and bits-per-byte metrics across diverse datasets like The Pile, with pronounced improvements noted in specialized datasets such as GitHub and EuroParl. Intriguingly, models that were not pretrained on The Pile showed more substantial benefits, indicating a promising avenue for smaller models to compete with larger architectures. The study also introduces a more memory-efficient retrieval implementation that significantly lowers RAM requirements, facilitating broader accessibility for researchers working with large-scale models. By evaluating the R1-Distilled-Qwen2.5-1.5B and confirming consistent gains across different model architectures, the findings underscore the robustness of nearest-neighbor TTT. This work not only enhances our understanding of language model adaptability but also highlights practical methods for implementing retrieval-augmented training, marking a notable stride in the pursuit of improving LLM performance in various contexts.