Evals before prompts: building an LLM OCR for KYC (blog.nilenso.com)

A recent initiative has leveraged large language models (LLMs) to automate the extraction of essential information from identity documents used in Know Your Customer (KYC) processes, significantly reducing costs and improving accuracy. Traditionally a manual and error-prone task, the initiative aimed to bring document verification in-house for a client, capitalizing on the potential of LLMs. A curated "Golden dataset" of 50 documents from various categories—such as PAN cards and Aadhaar—was developed to provide a benchmark for evaluating the system's performance, culminating in an impressive accuracy rate of 92%, while cutting costs by 90% compared to third-party services. The project included a meticulous evaluation process, establishing a rubric to assess the correctness of extractions based on various document-specific criteria. Iterative prompt engineering led to refined model interactions, emphasizing structured prompts, domain-specific context, and focused extractions per field. These developments not only enhanced the model's accuracy but also made processing more efficient, with latency dropping significantly. Key takeaways highlight the importance of foundational datasets and clear evaluation metrics in fine-tuning AI solutions, providing practical insights for future advancements in the AI/ML community.