Evaluating Genuine Reasoning in LLMs via Esoteric Programming Languages (arxiv.org)

Researchers have introduced EsoLang-Bench, a novel benchmark designed to evaluate the genuine reasoning capabilities of large language models (LLMs) using esoteric programming languages such as Brainfuck, Shakespeare, and others. Unlike traditional programming languages that dominate benchmark tests, these esoteric languages lack extensive public repositories and incentive structures, which allows for a cleaner assessment of reasoning without interference from memorization or prior training biases. The study reveals a stark contrast in performance, with top models showcasing proficiency in standard benchmarks (85-95%) but faltering dramatically on esoteric tasks, scoring between 0-11%. This finding is significant for the AI/ML community as it highlights the limitations of LLMs in true reasoning and adaptability, challenging the perception that high performance in mainstream tasks indicates genuine understanding or flexibility. The study also demonstrates that popular optimization techniques such as few-shot learning do not enhance the models' performance on these challenging tasks, suggesting that existing models may rely heavily on data memorization rather than authentic cognitive processes. EsoLang-Bench aims to mimic the human learning process, fostering the development of more capable AI systems that can genuinely understand and reason across diverse programming paradigms.