I built a programming language that LLMs can't use (blog.picheta.me)

A developer has created a new programming language called KillSwitch, designed specifically to challenge the code-writing abilities of large language models (LLMs) and coding agents. The language incorporates unique features that exploit LLM weaknesses, such as hidden lies in the specifications that humans can recognize but LLMs cannot interpret, strict rules on character counting for identifiers, and constructs that deliberately trigger LLM safety guardrails. As a result, KillSwitch serves as a testbed to explore the limits of LLM understanding and recall capabilities in a coding context. KillSwitch's effectiveness was evaluated through a series of benchmark tasks, revealing that while some LLMs performed relatively well—like Claude's Opus and the open-weight model MiMo-V2.6-Pro—none showcased true comprehension of the language. The tasks included simple commands like implementing a "Hello World" script or modifying code for temperature conversion, suggesting that although LLMs can adapt to the confusion, they do not genuinely understand the underlying code. As the developer plans to create even more complex tasks in the future, KillSwitch not only highlights the current limitations of LLMs but also invites further exploration into programming languages designed to test AI's cognitive boundaries.