What exists
An export format exists. No dataset has been published. There is no registry, no versioning, and no way to reach a file that has already left the machine. Unstarted.
The four export filters
Every export:
- the training verdict is allow
- no tombstone and no covering takedown selector
- no active quarantine
- the extracted text is present
Takedowns apply at export time, so a document stored after the takedown was recorded is still excluded.
Export record
The record shape
Field names, no values.
- Identity:
- the content hash as the record id, the URL, the canonical URL, and what this record supersedes.
- Content:
- the text, title, language, text format, and extractor version.
- Rights:
- the licence identifier and an attribution block: basis, licence URI, creator, copyright notice, source URI, modifications, and an explicit record of what is missing.
- Provenance:
- every URL the text was collected from, each with its own licence identifier, evidence hash, decision hash, decision reasons and collection time.
- Measurement:
- quality components, the language check, and the duplicate cluster.
Two URLs serving identical text export as one record carrying both source URLs and both decision chains.
A missing creator stays missing. No fallback to the site name, the domain or the page title.
Every difference between the response body and the exported text is named: extraction always, redaction and truncation when they happened.
Defaults
The three gates
| Gate | Default | What it protects against |
|---|---|---|
| Stale policy | refuses, overridable | The export says this text may be used now. Only as current as the last policy read. |
| Incomplete attribution | permits unless requested | The export cannot say who to credit. A defect once the file leaves. |
| Bypassed records | refuses by default | A file assembled from origins that said no. Overridable only by name, counted in the manifest either way. |
HMAC
Signing
The manifest signature is HMAC-SHA256, which is symmetric, so anyone who can verify a manifest can forge one. It proves the export came from a holder of the key. Nothing more. Asymmetric signing is unimplemented.
The algorithm is named in every manifest, so a verifier can refuse one. An unsigned manifest is reported as unsigned. With a key configured it is reported as a failure.
Never scored
Measurement
Quality components are stored separately and never summed into a score. The language field holds whatever the document declared, checked against a function-word comparison across eight languages. It flags disagreement. It identifies no language. Near-duplicates are labelled and removed only on request.