Gregory Kurtzer — creator of Rocky Linux, CentOS, and Apptainer — launched the OpenWALDO (Open Weights, Artifacts, Licenses, Data, Origins) project, which makes AI training data as verifiable as source code in Git repositories. The project provides a public corpus of 124 billion tokens and introduces an AI Bill of Materials standard for tracking data provenance.

image
image

What happened

Kurtzer launched OpenWALDO as an open-source project to create a shared and verifiable training data infrastructure. The public corpus includes 124 billion tokens, 75.1 million documents, and 20 collections, split into 1,051 shards with 18 license types. The project's key innovation is the AI Bill of Materials (BOM): an immutable document that tracks sources, licenses, cryptographic data hashes, and the full history from dataset to released model. Metadata is stored in Git with a mandatory Developer Certificate of Origin (DCO), while binary data is stored in a content-addressable storage in Parquet format. The commercial sponsor is CIQ (Ctrl IQ Inc.), also founded by Kurtzer, but governance remains with the community.

Context

The MIT Data Provenance Initiative documented a systemic problem with training data: over 70% of sampled datasets lack license information, and over 50% have licensing errors. The industry faces a black box: it is impossible to trace where data for model training comes from, what licenses apply to it, and how the data was transformed. Conceptually, the AI Bill of Materials is similar to SBOM/SPDX in software engineering — but adapted for ML data. Using Git with DCO for metadata and Parquet for binary data ensures a cryptographically verifiable change history and columnar storage efficiency. Kurtzer has a proven track record of creating sustainable open-source infrastructure: Rocky Linux became one of the most widely used Linux distributions after CentOS support ended.

Why this matters for the industry

OpenWALDO offers a common transparency standard: instead of every team scraping and collecting data from scratch, the industry gets a shared verified base with cryptographic hashes and a legally verifiable chain of provenance. This could become the foundation for regulatory compliance and reducing model collapse risks — model degradation when trained on synthetic data. For startups, the entry barrier to working with high-quality data drops sharply. With wide adoption, OpenWALDO will create a verifiable data provenance graph for the entire industry, enabling automated license compliance and building reproducible research pipelines from dataset to model. The BOM concept could become the de facto standard for describing provenance for open-source models — similar to how SPDX became the standard for SBOM.

Why this matters for users

This is the first project that makes training data verifiable like code on GitHub. OpenWALDO does not create models but builds the foundation beneath them: you can take the public corpus as a clean base, add your own data, and get a fully auditable stack. The public corpus is immediately usable as a source of clean, verified data for fine-tuning and RAG corpora. Repositories are available on GitHub (openwaldo/waldo-index and openwaldo/waldo), documentation is on openwaldo.org, and the community is on Slack. Researchers can start auditing licenses and sources of their existing datasets using BOM as a reference structure.

What is still unknown / limitations

The current 124-billion-token corpus is insufficient for pretraining modern SOTA-level models — practical value is currently limited to fine-tuning and RAG. The AI Bill of Materials itself does not make data legal: a BOM does not replace a full license audit. There is no information on data access speed, API latency, and storage throughput. A key future question is the speed of standard adoption by the community: integrations with Hugging Face and other registries, as well as tools for automatic BOM generation, will be needed. There is a risk of fragmentation if competing provenance standards start developing in parallel.

Sources

Author

Look at AI, editorial team