DPO just fixed text degeneration—why aren't we using it everywhere?
huggingface researchers applied direct preference optimization to ocr models, cutting text degeneration by 59.4% on average and proving dpo works for objective tasks beyond chat alignment.
huggingface introduced a methodology using direct preference optimization to mitigate text degeneration in ocr models, reporting a 59.4% average reduction in repetition loops across tested model families, with a peak reduction of 87.6%. unlike standard dpo applications for chat alignment, this approach treats objective task failures—such as degeneration loops versus correct transcriptions—as binary preference signals derived from the model's own outputs. this technique addresses a structural limitation in supervised fine-tuning, which optimizes token by token and often fails to penalize sequence-level errors, leaving a ceiling on reliability. by training on full completions, dpo explicitly labels degenerated outputs as incorrect, preventing the model from entering high-probability attractor regions that cause repetition. this research suggests dpo can serve as a broad tool for improving robustness in objective tasks beyond conversational ai.