Direct Preference Optimization Beyond Chatbots (huggingface.co)

In April, researchers announced the release of DharmaOCR, a specialized Optical Character Recognition (OCR) model focused on structured document extraction of Brazilian Portuguese text. The accompanying paper details the implementation of Direct Preference Optimization (DPO), a training stage that significantly reduces text degeneration—where the model repeatedly loops instead of accurately transcribing. Traditional supervised fine-tuning (SFT) achieved only limited improvements in degeneration rates, leaving a ceiling on its effectiveness. The DPO methodology, however, produced an average degeneration reduction of 59.4%, with a peak improvement of up to 87.6%, showcasing its potential as a direct tool for addressing this particular failure mode in structured tasks. The significance of this advancement lies not only in its immediate application to OCR but also in broadening the scope of DPO technology beyond chat alignments to objective tasks lacking human preference labels. By using the model's own degenerate outputs as negative training signals, DharmaOCR generated a robust preference signal derived from the specific failures of the model. This unique approach enables developers to tackle previously challenging issues in structured generation tasks without the need for extensive human annotation, providing a clear pathway for future enhancements in OCR and similar AI applications.