Fixing LLM Writing with Distribution Fine Tuning (rosmine.ai)

A new approach called Distribution Fine Tuning (DFT) has been introduced to address the common issues of repetitive and formulaic writing in large language models (LLMs). Traditional Supervised Fine Tuning (SFT) often fails to accurately capture the underlying distribution of training data, leading to overused phrases and lack of creativity. DFT significantly improves output quality, enhancing Maximum Mean Discrepancy (MMD) by 49% and Judge Model Quality (JMQ) by 63% compared to SFT. This leads to marked improvements in creativity (+164%), coherence (+28%), clarity (+16%), and meaningful detail (+146%), producing outputs that are indistinguishable from human-written text according to the Pangram AI detector. The technical foundation of DFT focuses on optimizing the model's output distribution rather than individual samples, allowing for a more accurate reflection of human writing styles. As shown with a 14 billion parameter model, DFT outperforms traditional methods on key metrics, suggesting its adaptability for various writing applications beyond the tested datasets like blogs and news articles. This breakthrough represents a significant advancement for the AI/ML community, with implications for future model training methods that prioritize quality of output distribution, potentially extending DFT’s impact to areas like AI-generated music and interactive content.