Truth is not a direction: a Tarski attack on LLM probes (abeljansma.nl)

Recent research has revealed significant limitations in the concept of "truth probes" for language models (LLMs), challenging the notion that these systems can embody a definitive direction of truth within their embedding spaces. A diagonal attack demonstrates that any probe attempting to evaluate truth for text inputs inevitably leads to paradoxes similar to those seen in Gödel’s incompleteness theorems and Tarski's undefinability theorem. The study shows that while probes may perform well on straightforward cases, they cannot universally capture the complexity of truth, especially when self-referential statements are involved. This exploration is critical for the AI/ML community, particularly in the realm of AI safety, as it highlights the limitations of relying on LLMs for determining factual truth. The findings suggest that while truth probes may aid in understanding model behavior and detecting misalignment, they cannot function as absolute truth-oracles. The implications extend beyond theoretical discussions, as they caution against users placing unwarranted trust in AI-generated truths. The research underscores the need for careful consideration of how AI models are perceived and utilized in decision-making contexts, ensuring that the potential for misinterpretation or over-reliance is recognized and mitigated.