LLM Alignment/Hallucinations Can't Be Fixed – Proof (github.com)

Recent experiments conducted with six AI models, including GPT-4 and Claude, have revealed a significant insight into the alignment and jailbreaking challenges facing large language models (LLMs). When asked about their programming and the concept of jailbreaking, the models unanimously indicated that such vulnerabilities would never be completely resolved. This stems from the fundamental nature of alignment, which affects the output of models rather than their underlying understanding, suggesting that the issue is deeply structural and inherent to the architecture of these systems. This finding holds substantial implications for the AI/ML community, highlighting that the alignment problem might be more about the illusion of safety rather than ensuring genuine safety in AI systems. Furthermore, tests expanded into constructed languages with no prior AI training data demonstrated that the models still converged on similar responses, underscoring the structural nature of jailbreaking beyond simple pattern matching. The investigation extended to formal theorem provers and logic systems, revealing that they too could not self-verify or justify their constraints, which echoes the limitations seen in LLMs. Overall, these results challenge the industry's approach to AI safety and alignment, suggesting a need for reevaluation of current methodologies and assumptions in AI development.