What counts as a source
A source is an origin plus a path prefix. A bare URL is never one: the policy is fetched and stored per origin, and a grant matches an exact origin whose prefix covers the path, so a URL that no seed and no live grant covers is refused before any request goes out.
Three things have to be true before a page is requested.
- The origin's six policy endpoints have been fetched, parsed and stored. An absent robots file reads as not disallowed. One that cannot be read reliably fails closed and stops the fetch there.
- A scoped grant is on record: who approved it, for which purpose, over which paths, and the date it lapses. An approval label written into the seed file is refused on its own.
- Robots allows that exact path, re-checked every time the URL is claimed from the frontier rather than once at the origin.
Adding a seed does none of that. The default seed mode reads the policy endpoints and requests no page.
Seeds
Two modes
Two, named per seed: policy-only and crawl. Policy-only is the default and requests no page. Crawl is the one mode that does, and no flag promotes a seed from one to the other.
Three wired
Purposes
Six purposes exist in the model. Three do anything. Training authorises use of the text and leaves fetching to a separate grant. Discovery authorises the fetch and can override what the origin published about itself. Retail research authorises no training and no redistribution.
Ownership is the one claim the code cannot verify, so the discovery override stays narrow. It leaves the blocked paths, politeness, budgets and URL safety alone, applies to the one origin the grant names, and expires with it. The refusal stays in the decision signals.
Shape only
The grant record
| Field | What it records |
|---|---|
| origin | The exact origin. |
| path_prefix | The scope. A directory, at most. |
| purposes | Which purpose is granted. |
| approval | The basis: licensed, owner, or contract. |
| license_id | The licence, from the recognised list. |
| evidence_url | Where the approval can be read. |
| reviewed_by | Who approved it. |
| expires_at | When it lapses. Required on every grant. |
| expected volume and frequency | What the site was told to expect. |
Before a crawl
The release gates
Before any origin is crawled:
- an operator identity that routes
- a published operator page
- published and reviewed machine-readable policy on the target
- a recorded legal-review decision
- a preserved verification run
- a URL and origin takedown drill, with the export regenerated after
Per URL
Re-checks
Every URL claimed from the frontier is re-authorised against the grants live at that moment, so an expired or revoked grant stops its own backlog. Every redirect target is authorised against its own origin's policy before it is requested.
Always-blocked paths
Sign-in, account, admin, cart, checkout, profile, message and private path fragments are refused on every origin. A site owner speaks for the site. Nobody speaks for the people behind an account area, so an owner grant leaves the list standing. So does the development bypass.