Llama-Factory: Unified, Efficient Fine-Tuning for 100 Open LLMs (github.com)

LLaMA-Factory is an open-source fine-tuning framework that unifies and automates efficient training across 100+ open LLMs (LLaMA family, Qwen/Qwen-VL, Gemma, Mistral/Mixtral, Phi, DeepSeek, ChatGLM, etc.). It bundles end-to-end recipes for pre-training, supervised fine-tuning, reward modeling and RL (PPO, DPO, KTO, ORPO, SimPO), and supports full-, freeze-, LoRA- and QLoRA-style workflows (including OFT/QOFT), plus multimodal fine-tuning for vision/audio/video tasks. The project publishes tutorials, Colab notebooks and platform integrations (AMD/ROCm, SageMaker, vLLM/SGLang inference backends) and an arXiv paper documenting its approach. Technically, LLaMA-Factory emphasizes practical, scalable performance: FSDP+QLoRA enables 70B tuning on 2×24GB GPUs; it supports multi-bit quantization (2–8 bit) through AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ, FlashAttention-2, Liger kernel, RoPE scaling, dataset streaming, and experiment tracking (Wandb, TensorBoard, LlamaBoard). It integrates advanced optimizers and algorithms (GaLore, BAdam, APOLLO, Adam-mini, DoRA, LongLoRA, Mixture-of-Depths, LoftQ, PiSSA) and offers fast inference options (vLLM with up to ~270% speedups, Gradio UI, API). For practitioners, it lowers the barrier to reproduce SFT/RLHF/multimodal workflows across diverse models and hardware, making efficient, production-ready adaptation of large open models more accessible.