PrivacyBench: An open benchmark for de-identifying text that scores synthesis (huggingface.co)

PrivacyBench has been introduced as an innovative benchmark aimed at enhancing the de-identification of semi-structured data, particularly in text exports from tools like email and Slack. This benchmark allows researchers to assess both the detection of personally identifiable information (PII) and the coherence of synthetic replacements through novel metrics. The benchmark comprises synthetic data exports based on 21 distinct personas, generated using the Fabricate tool, with ground truth labels created without human annotators. Moreover, the benchmark demonstrated that employing Tonic Textual for named entity recognition (NER) significantly improves PII recall compared to large language models (LLMs). The significance of PrivacyBench lies in its dual focus on privacy and utility within synthetic data, addressing a critical challenge in AI/ML: the balance between accurately detecting sensitive entities and effectively replacing them without compromising the coherence of the data. The evaluation of various synthesis pipelines showed that the combination of Tonic Textual for NER and the Opus 4.8 LLM produced the highest scores for synthesis accuracy and overall pipeline performance. This represents a substantial advancement in creating synthetic data that remains useful while ensuring privacy, reducing costs by over 60% compared to LLM-only approaches. The benchmark's code and datasets are readily accessible on GitHub, encouraging further research and innovation in the field.