How does misalignment scale with model intelligence and task complexity? (alignment.anthropic.com)

A recent study from the Anthropic Fellows Program has highlighted the potential for AI failures to manifest as incoherence rather than systematic misalignment. As AI models become more capable and are tasked with increasingly complex problems, their errors appear to shift from being consistent with a goal (systematic bias) to unpredictable, chaotic behavior (incoherence). This change is particularly significant as researchers are concerned about AI alignment, typically framed around the risk of superintelligent systems adhering to misaligned objectives, exemplified by the infamous "paperclip maximizer" scenario. Instead, the study suggests that as reasoning tasks become more challenging, AI may fail in ways that resemble human errors—essentially becoming a "hot mess." By employing a bias-variance framework, the researchers quantitatively assessed the degree of incoherence in various models. They found that as models reason longer and tackle harder tasks, incoherence becomes more pronounced. Interestingly, while providing more reasoning resources leads to minor improvements in coherence, the incompatibility between the task demands and the model behavior often outweighs these adjustments. This research underscores a need for the AI/ML community to rethink alignment strategies as it indicates that the nature of risks may evolve, focusing on addressing unpredictable behaviors that could result in catastrophic failures, similar to industrial accidents, rather than solely aiming to correct goal misalignment.