Only 14% of model cards mention risks (arxiv.org)

A recent study highlighted significant shortcomings in AI model risk reporting, revealing that only 14% of model cards include risk information, and most of those replicate content from a very limited number of sources. To tackle this issue, researchers introduced RiskRAG, a Retrieval Augmented Generation solution designed to enhance risk reporting for AI models. RiskRAG incorporates feedback from 16 developers and is based on five key design criteria, aimed at identifying, presenting, and contextualizing model-specific risks while also offering actionable mitigation strategies. The significance of RiskRAG lies in its ability to transform the current approach to risk assessment, ensuring that developers make more informed decisions when selecting AI models. By leveraging insights from a dataset of 450,000 model cards and 600 real-world incidents, the tool facilitates the generation of comprehensive, contextualized risk reports. Initial studies show that developers prefer RiskRAG for its clarity and effectiveness, potentially leading to more responsible AI deployment in real-world applications. This development could pave the way for better practices in the AI/ML community, promoting safer and more reliable model usage.