Does Opus 4.6 find the needle in the haystack? (georggrab.net)

Claude Opus 4.6 has undergone testing to evaluate its capability to extract previously unseen information using its extended 1 million token context length. By substituting spell names from the first four Harry Potter books with random Latin-sounding names, the experiment aimed to measure the model's ability to retrieve information that was not included in its training data. The results were striking; when source names were altered, the model achieved a recall rate of 0%, failing to retrieve any spells, while it performed with a recall rate of 90% for the original dataset, accurately identifying 25 of 27 spells. This testing underscores the significant challenge in using AI models for retrieval of information outside their training datasets, highlighting that even advanced models like Opus 4.6 leverage extensive pre-training knowledge. The findings illustrate the limitations of context-based retrieval and suggest that improvements are necessary for AI systems to better handle novel inputs. Overall, the results serve as a cautionary tale for AI researchers and developers, emphasizing the need to consider training data biases and the true capacities of models when designing retrieval tasks.