User-agent

Published identity

CrawlCharter/0.1 (+https://crawl.devaesia.com/bot/; mailto:crawler@devaesia.com)
  • Product token: CrawlCharter
  • Operator: DEVAesia, Romania
  • Contact: crawler@devaesia.com. This address is also sent in the From header of every request.

Operating status

Research only

Not released. It has only crawled sites DEVAesia owns and has collected from no third-party site.

Nothing it collected has trained a model, and no dataset has been published.

What it is doing here

Measuring what publishers declare about AI use, and testing whether a crawler can collect only what it has evidence it may collect.

What it requests

  • GET only. No form, no login, no cookie, no JavaScript, no link off your site.
  • One request at a time per site, at least two seconds apart, slower if your Crawl-delay, a 429 or a 503 says so.
  • HTML and plain text only. No images, video, scripts, stylesheets, fonts or downloads.
  • Six fixed policy files: /robots.txt, /.well-known/tdmrep.json, /.well-known/ai.json, /.well-known/ai.txt, /ai.txt and /llms.txt.
  • Pages inside a path an operator has approved.

A run has a page cap and a depth cap. Per-origin request and byte budgets ship unset, and the tool reports them as unset.

What it honours

  • robots.txt, per RFC 9309, including Allow precedence and Crawl-delay.
  • The well-known AI policy files, in both the JSON and the text form.
  • The W3C TDM Reservation Protocol, in the site file, in response headers, and in HTML metadata.
  • Content-Usage and Content-Signal directives, in response headers and in robots.txt.
  • The noai family of tokens in HTML metadata.
  • The legacy /ai.txt as a refusal only.

What it keeps

  • The policy files themselves and a hash of each, so a decision can be replayed later without asking your server again.
  • For pages inside an approved path: the URL, the extracted text, the decision, and the evidence behind it.
  • Response metadata: status, timing, content hash.

Raw bodies and the text of anything it may not use are off by default. Detected personal data is redacted before storage. A detected credential drops the page and keeps no text.

Retention periods are not set.

Asking it to stop

Any of these works.

  1. Email crawler@devaesia.com. Name the URL or the domain.
  2. robots.txt. A Disallow rule for the CrawlCharter user-agent, honoured on the next read.
  3. An AI policy file. The well-known JSON or text form, or a TDM reservation. Training use only.

Response target: acknowledged within one working day. What each one does to content already collected: Owner controls.

Urgent contact

If something goes wrong

If this crawler causes load you did not expect, or requests something it should not have: email crawler@devaesia.com, marked urgent.

The user-agent string, the times and a few paths help. It stops either way.