Do Code Models Suffer from the Dunning-Kruger Effect? (arxiv.org)

A new paper examines whether code-focused large language models display a Dunning–Kruger–like effect (DKE)—overconfidence from limited competence—by comparing model confidence and real performance on coding tasks across many programming languages. The authors find that state-of-the-art code models do mirror human overconfidence: models are most overconfident when they are less competent overall and when operating in unfamiliar or low‑resource languages. In other words, weaker models and language settings with sparse training data exhibit the strongest mismatch between predicted confidence and actual success, showing a clear relationship between competence and the strength of the DKE-like bias. This result matters for the AI/ML community because it reframes calibration and trust concerns as cognitive-style biases that vary by domain and data coverage, not just model size. Practically, it raises risks for coding assistants (silent hallucinations, misplaced trust in incorrect suggestions) and argues for prioritized fixes: better uncertainty quantification and calibration, abstention or clarification interfaces, more diverse training/evaluation across rare languages, and benchmarks that measure confidence–performance alignment. The study underscores that improving raw capability isn’t enough—models must also know when they don’t know, especially in low-resource coding contexts.