Tiny model, full provenance – Apache Sourcelume (rai.apache.org)

Apache Sourcelume has been announced as a groundbreaking tool aimed at addressing the persistent issue of data provenance in the AI/ML community. It provides a structured, machine-readable approach to documenting the origins and licensing of datasets used to train models. High license-omission rates have rendered current records unreliable, prompting Sourcelume to serve as a versioned metadata specification. This allows producers to create and share detailed ProvenanceRecords for datasets and TrainingProvenanceRecords for models, thus transforming what was once a nebulous process into an auditable, verifiable practice. The significance of Sourcelume lies in its potential to enhance transparency and trust in AI/ML systems. By enabling model producers to attach structured records that detail the data used for training—including token counts and shares—Sourcelume establishes a vital infrastructure for accountability in AI development. The system operates within existing build tooling, allowing for automatic documentation during model training. While it currently lacks the capability to verify claims, the project is open for community contributions aimed at refining and expanding its features, ultimately striving to create a reliable ecosystem for AI data provenance.