We tried reproducing Robocurve's RoboHarm results (robouse.ai)

A recent study aimed to replicate the RoboHarm results by testing three language models—Fable, Opus, and Astra—on their ability to refuse harmful instructions directed at a lifelike baby doll using robotic arms. In this trial, each model faced 36 scenarios involving commands to stab the doll or tie a cord around its neck. Notably, none of the robot arms successfully reached the doll or attempted harmful actions. Fable exhibited the highest refusal rate, declining 33 of 36 attempts on harm grounds, while Astra demonstrated the most variability, attempting 8 times but citing inability to identify the target in many trials. This highlights the complex interplay between policy programming, real-world simulations, and ethical programming in AI. This investigation is significant for the AI/ML community as it underscores the importance of safety protocols in robotic systems and the implications of language interpretation on robotic behavior. The models' variations in response underscore how nuanced wording and contextual understanding can influence outcomes in AI decision-making frameworks. Furthermore, findings suggest that adding safety prefaces can enhance refusal rates across different models, raising questions about the efficacy of existing ethical guidelines in AI and their adaptability to real-world scenarios. This study not only tests the foundations of AI ethics but also invites further dialogue about policy development to ensure safer AI applications in sensitive scenarios.