A Red-Team Study of Anthropic Fable 5 and Opus 4.8 Models (arxiv.org)

A recent red-team study has evaluated the adversarial robustness of Anthropic’s large language models, Fable 5 and Opus 4.8, against a range of automated jailbreak attacks. Utilizing the HackAgent framework, researchers executed hundreds of thousands of adversarial attempts, covering 7,826 harmful intents across a comprehensive ten-category harm taxonomy. Although both models demonstrated a commendable ability to resist the majority of attacks, they remain vulnerable, particularly to adaptive iterative attacks, indicating a broader residual risk than suggested by aggregate performance metrics. The findings reveal that while Fable 5 exhibits a lower failure rate at 6.1% under the worst-case scenarios, Opus 4.8's failure rate stands at 11.5%. Alarmingly, even with their enhanced defensive configurations, both models produced confirmed harmful outputs—1,620 for Opus 4.8 and 702 for Fable 5. This underscores a critical takeaway: the most advanced models are still susceptible to persistent automated adversarial pressure, raising concerns about their deployment in sensitive applications. The study highlights the importance of ongoing research into the robustness of AI systems as adversarial techniques evolve.