Engineering an LLM-Based Data Classifier (getnumberseven.com)

A new LLM-based data classifier has been engineered for Ethyca's Fides platform, significantly improving the accuracy of sensitive data categorization. Developed over six months, this classifier achieved over 80% accuracy using a quantitative evaluation framework, and more than 95% accuracy on easier benchmarks, leveraging only metadata to classify data without direct access to sensitive information. This focus on metadata addresses security concerns in enterprise environments while maximizing the potential of LLMs beyond traditional methods that rely on brittle regular expressions or extensive training datasets. Key advancements include a robust evaluation process that intertwined human labeling with AI outputs, demonstrating that LLMs can surpass human labeling accuracy when combined strategically. Moreover, the classifier was capable of operating on models as small as 32 billion parameters, achieving classification rates of 95 fields per minute at low operational costs. This innovative approach not only streamlines the deployment of data classification tools but also highlights the transformative potential of AI in enhancing data governance practices within enterprises. The project sets the stage for future enhancements, including cost optimization and potential fine-tuning to further leverage LLM capabilities.