Extract-0: A specialized language model for document information extraction (arxiv.org)

Researchers introduce Extract-0, a 7-billion-parameter language model fine-tuned specifically for document information extraction that the paper claims was produced for roughly $196. Using a memory-preserving synthetic data pipeline to generate 280,128 diverse training examples, parameter-efficient supervised fine-tuning with LoRA (changing just 0.53% of weights — 40.4M of 7.66B parameters), and a reinforcement-learning stage called Group Relative Policy Optimization (GRPO), Extract-0 achieves a mean reward of 0.573 on a 1,000-task extraction benchmark. That score outperforms much larger or more general models in the paper’s comparisons: GPT-4.1 (0.457), o3 (0.464), and GPT-4.1‑2025 (0.459). The work is significant because it demonstrates that targeted data-generation plus lightweight fine-tuning and a semantic-similarity reward can yield specialist models that beat general-purpose LLMs on niche tasks while using far less compute and storage. Key technical takeaways are the use of synthetic, memory-aware data to preserve extraction targets, LoRA for parameter-efficient adaptation, and a novel semantic reward to handle ambiguity in labeled extraction outputs. This suggests practical routes to cheaper, higher-performing production systems for structured document parsing and points to broader trade-offs between model size and task-specific optimization.