The six endpoints
| Path | Mechanism | Maturity | Can deny | Can allow |
|---|---|---|---|---|
| /robots.txt | RFC 9309 Robots Exclusion Protocol | Published RFC | fetch | fetch only |
| /.well-known/tdmrep.json | W3C TDM Reservation Protocol | Community Final Spec | training | no, alone it yields review |
| /.well-known/ai.json | Well-known AI policy, JSON form | Individual Internet-Draft | fetch and training | training, only paired with a recognised licence |
| /.well-known/ai.txt | Well-known AI policy, text form | Individual Internet-Draft | fetch and training | training, only paired with a recognised licence |
| /ai.txt | Legacy root path | Community mechanism | training | never |
| /llms.txt | Navigation convention | Community proposal | no | no |
A policy file has to parse as one. A host that answers every path with its HTML shell makes all six look present, and HTML parsed as robots.txt yields no rules, which reads as allow everything. That fails closed and gets its own endpoint state.
This site publishes five of the six. /ai.txt is absent on purpose. It can only refuse, and it cannot carry a licence field.
Signals
Thirteen signal kinds, each with a precedence integer. Every one that contributed is stored with the decision. Robots access sits at the bottom and risk findings at the top, above the development bypass. A planted credential is a safety problem.
Two verdicts
A fetch denial forces a training denial, with the reason recorded.
The model carries review on either axis. The engine only reaches it on training, so fetch resolves to allow or deny and seven of the nine cells are reachable.
What review means
Review means the evidence stopped short. A big review pile means bad source picks or bad review tooling.
The four recognised licences
Four licence identifiers are recognised, resolved from canonical deed URLs. Anything else is returned verbatim, fails the allowlist, and routes to review.
Third-party content, attribution, share-alike, database rights, contract terms and jurisdiction stay open. A parser cannot answer those.
Two hashes
The evidence hash covers what the origin said. A 304 and a 200 with the same bytes hash the same, so drift does not fire when nothing changed.
The decision hash adds the engine revision, the parser versions, the product token and the licence allowlist. In the evidence hash, parser versions would make a parser upgrade look like every publisher changing policy at once.
Re-reading and drift
Policies get re-read and the difference is classified. A more restrictive change quarantines what was collected under the previous policy. A more permissive one releases nothing automatically.
An origin whose policy could not be re-read is reported and left alone.
Hard limits
- GET only. No form submission, no state-changing request.
- No JavaScript execution and no headless rendering.
- No cookies sent, and outbound cookies stripped.
- No authentication, and no paywalled or credentialed source.
- No cross-origin discovery.
- No image, audio, video or binary retrieval.
- No interpretation of terms of service.
- No language model anywhere in the decision path.