A model is what it eats. Now prove it.

Every duplicated document, boilerplate page, licence-ambiguous scrape, or contaminated benchmark answer that reaches a training run burns GPU-hours teaching the model nothing — or worse, teaching it something you cannot defend. For a resource-constrained sovereign program, corpus composition is not a preprocessing chore; it is one of the largest cost and quality levers that exists.

Most teams treat data preparation as ad-hoc scripts run once and forgotten. Nothing is measured, nothing is repeatable, and when a regulator, customer, or court asks what exactly did this model learn from? — there is no answer.

Shuddhi makes the corpus an engineered, audited product: a staged pipeline with a measured report at every stage, and a sealed manifest at the end that answers that question precisely.

Provenance gate — every shard enters tagged with source, licence, and data class. Untagged material is refused. Customer data is excluded mechanically at this stage: the pipeline will not accept it, so no downstream mistake can train on it.

Language identification — the Indic/English mix is controlled deliberately, not inherited accidentally from whatever was scraped.

Deduplication — exact and near-duplicate removal, so the model never pays twice to read the same page.

Quality scoring — coherent, information-rich material is prioritised; boilerplate, spam, and junk are scored out.

Domain balancing — tokens are allocated across coding, reasoning, Indic languages, and regulated domains by decision, with the mix reported.

Contamination check — the corpus is screened against our evaluation batteries, so a benchmark result can never be an echo of training data. This protects the integrity of every number we publish.

The build manifest — the accepted corpus is sealed under a content hash with its full composition report: tokens by domain and language, dedup rate, quality distribution, and every exclusion counted by reason.

Shuddhi processed our full 176.1 GB from-scratch training corpus — 33,047,370 documents across 15 languages, all CC-BY-4.0 licensed — end to end on one 2-vCPU CPU box. Zero GPU, zero data egress.

The checklist is closed: every filter now runs on the whole corpus, not a sample. Shuddhi's chain — exact and near-duplicate removal, quality scoring, perplexity screening, a toxicity pass, contamination checking, and PII redaction — is applied and receipted end to end.

The filtered build kept 32,289,800 of 33,047,370 documents (97.71%), every drop counted by reason: 127,024 exact duplicates (0.38%), 272,640 near-duplicates (0.83%), 345,505 perplexity outliers (1.05%, a p99 cutoff by design), 7,086 low-quality (0.02%), and 5,306 toxicity (0.016%) — the last a conservative lexicon tier that removes the unambiguous tail, not a comprehensive safety classifier; that tier is on the roadmap.

Contamination: zero — every one of 33,047,370 documents checked against our evaluation sets, not a sample, so the benchmark numbers we publish measure capability, not memorisation. 182,781 PII spans redacted on emission (email, phone, Aadhaar, PAN, Luhn-checked cards, IP). Tokens stay an estimate — ~45.9 billion (derived) under our own tokenizer — which is why the headline figures are byte- and document-denominated (176 GB, 33 million documents): a token count shifts with the tokenizer, the corpus does not.

The result is sealed under a filtered build hash chained to the source-corpus hash, the exact filter config, and the tokenizer — the config pins the *tools*, not just the thresholds: the near-duplicate droplist, the toxicity lexicon, and the tokenizer are all hashed into the receipt, so every training run cites corpus, filters, and tokenizer together in its ledger. Because the pipeline is partition-invariant, builds parallelise without changing the receipt.

The 84,275-copy near-duplicate cluster described earlier surfaced here, in the English shard (2.4% near-duplicates) — not where an earlier sampled pass had suggested. Sampling hinted; the full-scale receipt corrected it. That is precisely why filters have to be *applied* at scale, not estimated.

Open-core products usually leave this vague, and that vagueness is what makes people hesitate to adopt them. So, plainly:

Nothing is ever removed from the open engine. Buying does not unlock a feature that was disabled — the paid tiers add a *service* alongside the engine. If a commercial capability needs an engine change, that change ships in the open engine too, because it is a plugin-API improvement and everyone gets it.

You hit the line when you need to prove something to someone else. Running the pipeline, generating receipts, drafting your Article 53 summary and attesting a corpus are free and unlimited, forever, for any size of corpus. What is paid is the organisational layer: a signed ledger others can verify, a review queue several people share, and reporting across many builds over time.

Your receipts stay yours, and stay verifiable without us. A build hash is computed by the open engine from your corpus, so anyone can recompute and check it with the free tool alone. If you stop paying — or never start — nothing you already produced becomes unreadable, and no artefact is held hostage. The hosted registry adds signatures, lookup and retention; it is not a decoder ring for your own data.

No telemetry, no account, no phone-home. The open engine does not contact us, ever. We do not know who is running it or what they are cleaning, and we would rather keep it that way than trade it for a funnel metric.

Article 53(1)(d) of the EU AI Act requires every provider placing a general-purpose model on the Union market to publish a sufficiently detailed summary of training content, on the template the Commission's AI Office issued in July 2025. Enforcement powers became active on 2 August 2026. Models already on the market must publish by 2 August 2027.

Shuddhi emits that summary as a build artefact, because it already holds the inputs: source, licence, data class, language and acquisition date are required registry fields — a shard missing any of them is refused, not defaulted.

``` factory.py report --registry corpus.json --eu-ai-act --manifest BUILD-MANIFEST.json ```

One field is worth dwelling on. Where every other provider must *describe* how they handled user data, the honest answer here is that a build containing it cannot be produced — and the refused shard appears in the generated summary, with its reason. The refusal shows up in the filing, not just in a README.

The report is a draft for your counsel, never a compliance claim: compliance is a legal position a provider holds, not a property of a tool. Anything the corpus cannot know — provider identity, crawler behaviour, text-and-data-mining opt-out policy — is printed as an explicit gap rather than guessed at. An incomplete summary you can see beats a confident one that is wrong.

India. The DPDP Act made personal data in a training corpus a board-level question, and there is no grandfathering for a model already shipped. Shuddhi refuses customer data structurally rather than filtering it hopefully, redacts personal data on emission with detectors built for Indian identifiers — Aadhaar in its printed grouping, PAN, Indian mobile numbers, Luhn-checked cards — and runs entirely inside your perimeter, CPU-first, so no corpus leaves your infrastructure to be cleaned.

European Union. Article 53(1)(d) of the AI Act requires a published summary of training content on the Commission's template. Enforcement powers activated on 2 August 2026; models already on the market must publish by 2 August 2027. Shuddhi emits that summary directly from the registry and build manifest, and marks every field a corpus cannot know as an explicit gap for your counsel rather than guessing at it.

United States and elsewhere. No statute yet compels a training-content disclosure, but litigation and enterprise procurement already do. "What was this model trained on, and can you show me?" is now a standard question in a vendor security review and a predictable one in discovery. A build hash chained to a filter configuration is a far better answer than a recollection.

The useful part is that these are not three products. It is one receipt, read three ways — because provenance, once recorded properly, is regime-agnostic. Record it once at build time and the regulatory framing becomes a rendering problem rather than a data-collection emergency.