Show HN: PDF 2 Context – Convert PDF text to JSONL files (github.com)

A new command-line interface tool, PDF 2 Context, has been introduced to convert directories of PDF files into structured JSONL context files, enhancing their utility in large language models and retrieval-augmented generation (RAG) pipelines. This production-quality tool features recursive PDF discovery and text extraction that maintains the document's original layout through pdftotext. It also integrates automatic optical character recognition (OCR) using ocrmypdf and Tesseract for PDFs with low text yield, ensuring robustness in handling both standard and scanned documents. The significance of PDF 2 Context lies in its ability to streamline the preprocessing of PDF data, which is crucial for training AI models. With configurable options, users can customize overlapping word chunks, parallel processing with multiple workers, and define processing timeouts for OCR, making it adaptable to various needs. The output includes per-file JSONL context files and an overarching statistics manifest, supplying a clear overview of the processing results. This tool ultimately facilitates easier integration of rich PDF content into AI workflows, promoting advancements in natural language understanding and machine learning applications.