Can LLMs SAT? (blog.aiono.dev)

Recent testing evaluated the reasoning capabilities of large language models (LLMs) by applying them to SAT (satisfiability) problems, which require consistent application of logical rules. The findings reveal significant limitations in LLM reasoning: while models like GPT-5.2 performed reasonably well on simpler problems, they struggled with larger SAT instances, often producing invalid assignments or making erroneous claims about satisfiability. The analysis indicates that as the complexity of SAT problems increases, LLMs tend to perform like random guessers, pointing to a decline in reasoning competency. Testing involved generating random SAT instances in Conjunctive Normal Form (CNF) and assessing LLM outputs against a standard SAT solver. Despite the advancements in LLM capabilities, the results suggest that these models cannot reliably reason through complex logical tasks, which could have implications for their application in critical domains that require rigorous logical reasoning. This research underscores the importance of incorporating safeguards or additional validation steps when deploying LLMs for tasks demanding high levels of reasoning accuracy.