Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Models (bytedance-seed.github.io)

Stable-DiffCoder has been unveiled as an advanced code diffusion Large Language Model (LLM), constructed on the Seed-Coder framework and designed to enhance code quality through innovative training techniques. The model introduces a unique block diffusion continual pretraining (CPT) phase, paired with a tailored warmup strategy and a block-wise clipped noise schedule. Remarkably, when utilizing only CPT followed by supervised fine-tuning (SFT), Stable-DiffCoder outperforms many existing code models, indicating that diffusion-based training can yield superior results compared to traditional autoregressive (AR) methods, particularly within stringent data and architectural limitations. This development is significant for the AI/ML community as it highlights the potential of diffusion-based training in refining code models, overcoming the noise issues often associated with traditional bidirectional training. Through strategic initialization from an AR checkpoint and the implementation of two critical designs addressing gradient instability during training, Stable-DiffCoder achieves impressive robustness, leading to superior performance across its Base and Instruct versions. These advancements not only set a new benchmark for code modeling but also pave the way for future exploration of diffusion methodologies in LLM training, demonstrating their capability to leverage clean knowledge for improved outcomes.