DeepFabric – Generate High-Quality Synthetic Datasets at Scale (lukehinds.github.io)

DeepFabric is a new library for generating high-quality synthetic datasets specifically aimed at language-model training, evaluation and research. Instead of ad-hoc prompt-by-prompt sampling, it builds a structured conceptual map of a domain—either hierarchical topic trees or experimental topic graphs—and uses that topology to drive systematic, context-rich example generation. That approach helps create broader coverage and more consistent quality across datasets, which matters for tasks like model distillation, agent evaluation, domain-specific fine-tuning and benchmark construction. Technically, DeepFabric runs a three-stage pipeline: topic generation (tree or graph), dataset generation driven by those topics (using LLMs to expand topics and produce examples), and packaging into standard formats ready for immediate use. It offers a YAML configuration format, a Python API mirroring CLI commands, and integrations with OpenAI, Anthropic, local Ollama instances and cloud providers. Generated artifacts can be exported to the Hugging Face Hub with auto-generated dataset cards and metadata. Practical tooling includes deepfabric validate, visualize and upload to support complex workflows and reproducible dataset creation. The result is a scalable, configurable way to produce diverse, domain-aware synthetic corpora that are easier to audit, reproduce and integrate into ML pipelines.