Access

  1. It identifies itself. A product token, an operator URL and a routable From address on every request. A placeholder or an unroutable address refuses to start the crawl.
  2. It asks before it takes. RFC 9309, longest-match precedence, observed Crawl-delay. An unreadable robots file fails the origin closed.
  3. It stays where it was sent. Same-origin by default. A cross-origin redirect target is authorised before the redirect is requested.
  4. It never reaches past a control. No authentication, no cookies, no CAPTCHA handling, no paywall bypass, no identity rotation, no form submission, no state-changing request.

Use

  1. Fetch and use are separate verdicts. Every URL carries both, and permission on the first grants nothing to the second.
  2. Silence resolves to review. An unknown rights position is unusable. Evidence is the only thing that moves it.
  3. An allow must be attributable. Training allow needs an explicit permission paired with a recognised licence, or an active scoped grant. Four licence identifiers are recognised.
  4. Conflicts are preserved. Every contributing signal is stored with the decision. Explicit denial wins, and nothing upgrades to allow.

Impact

  1. It crawls slower than the site would allow. One request at a time per origin, at least two seconds apart, slower when a Crawl-delay, a 429 or a 503 says so.
  2. It stores as little as it can. Raw bodies and review text off by default. No images, scripts, stylesheets, fonts, video or downloads. Personal-data indicators redacted before storage.

Reversal

  1. A refusal reaches content already collected. Re-reading a policy that has become more restrictive quarantines what was collected under the old one.
  2. A takedown deletes. The exclusion is written before the deletion, and it applies at export time, so content stored after the takedown is excluded too.

What would stop this project

The hypothesis can fail. If the published signals are too fragmented to decide training automatically, this stays an observatory and an allowlisted collector.

Work stops on any of these:

  • an unapproved origin is fetched
  • a robots or rate-limit violation repeats
  • an allow decision cannot be reproduced from its stored evidence
  • a test record cannot be deleted from current exports
  • secrets or personal data reach an allowed export
  • a parser ambiguity defaults to allow
  • someone pushes to bypass a publisher control

The licence on this page

This charter is published under CC-BY-4.0. It is the one path on this site that resolves to training allowed. Every other path here reserves its rights.