PatentScore: Multi-Dimensional Evaluation of LLM-Generated Patent Claims (aclanthology.org)

A new evaluation framework named PatentScore has been introduced to assess patent claims generated by large language models (LLMs), addressing a significant gap in the reliable evaluation of high-stakes texts. Traditional natural language generation metrics struggle to capture the complexity and precision required for legal documents like patents, which necessitate rigorous structural and semantic evaluations. PatentScore innovatively incorporates a hierarchical decomposition of claim elements and validation grounded in legal and technical standards, enabling a robust scoring system across multiple dimensions. In experiments conducted with a dataset of 400 patent claims, PatentScore demonstrated a correlation of 0.819 with expert annotations, significantly surpassing existing NLG metrics. This performance positions PatentScore as a new benchmark in the evaluation of LLM-generated patent claims, underscoring its potential to enhance the reliability of AI applications in legal contexts. The introduction of this framework could pave the way for advancements in patent generation and validation, ultimately fostering more accurate and legally sound AI-generated documentation in high-stakes environments.