We don't let the LLM decide what's clinically allowed (www.hamo.ai)

A new approach to integrating large language models (LLMs) into therapy has been developed, emphasizing the importance of restricting decision-making abilities to ensure safety and efficacy. The framework involves a structured pipeline where the LLM is only responsible for assessing input and generating output, while critical clinical decisions are determined by a deterministic Python code. This design prevents the model from improvising therapeutic techniques, thereby minimizing the risk of inappropriate, emotionally-driven responses during therapy sessions. The significance of this development lies in its ability to enhance clinical safety by ensuring that therapy techniques are applied based solely on a client's state, as defined by their clinical score. This innovative pipeline includes a multi-step process for crisis detection and risk assessment, enabling a quick response if a client poses an acute risk. The structured approach limits the model's conversational flexibility but prioritizes patient safety, demonstrating a clear trade-off between creative language generation and clinical decision integrity. While it currently lacks the ability for general conversation and comes with increased engineering complexities, this method marks a crucial advancement for safely deploying AI in therapeutic contexts.