Benchmarking LLMs on whether they can play FizzBuzz (github.com)

A new benchmark has been introduced to assess the performance of large language models (LLMs) in playing the children's game FizzBuzz, which tests their arithmetic skills, counting ability, and generalization from training data. The benchmark allows for customizable rules, challenging LLMs to respond accurately under different conditions, such as varying the numbers for "fizz" and "buzz." This simple exercise highlights whether models can merely rely on learned patterns or genuinely generalize to new scenarios. Initial results have shown that recent models, particularly from OpenAI, struggled with the standard FizzBuzz version, often performing poorly compared to Claude models, which excelled in both standard and modified versions. The testing revealed that most LLMs, despite having robust capabilities, fail to generalize effectively when faced with slightly altered rules, raising questions about their adaptability in multi-turn conversational contexts. This benchmark not only provides insights into specific weaknesses of existing models but also identifies pathways for enhancing their training and performance in more complex tasks.