Request lifecycle

From crawler claim to origin decision.

WARD is a gateway pattern: inspect the requester, resolve a public policy, enforce before delivery, and record exactly what the origin decided. Each layer exposes its uncertainty.

HTTP request→Identity→Policy→Decision→Response + receipt

The four layers

Identity

What evidence supports the crawler name?

A known User-Agent starts as claimed. A deployment can supply provider IP-range, reverse-DNS, or trusted-proxy evidence to produce verified or mismatch. WARD never upgrades a bare string to verified.

Policy

Which uses are permitted on this path?

/.well-known/ward.json declares defaults and path overrides for training, search indexing, and user-triggered fetches. The most specific matching rule becomes the decision input.

Enforcement

What must happen before bytes leave the origin?

A server policy returns allow or redact; an advisory policy returns observe. On protected Next.js routes, values are omitted during Server Component render so they do not leak through the React flight payload.

Receipt

What can another system reproduce?

Response headers and the optional event API carry the identity status, represented uses, action, matched rule, policy version, policy hash, and event id. Measured redaction is recorded only by adapters that can actually count it.

Policy and marked content

Policy controls whether the marked block is delivered to the requester represented by a known crawler identity.

ward.json

{
  "version": "1.0",
  "enforcement": "server",
  "default": {
    "indexing": true,
    "training": false,
    "summarization": true
  },
  "rules": [{
    "path": "/ward-canary",
    "redact": true
  }]
}

Server-rendered component

{wardAction !== "redact" && (
  <div data-ward="redact"
       data-ward-reason="synthetic-canary">
    ...synthetic fixture...
  </div>
)}

Crawler names represent different uses

Example	Use	Policy field
GPTBot / ClaudeBot	`training`	`training`
OAI-SearchBot / Claude-SearchBot	`search-index`	`indexing`
ChatGPT-User / Claude-User	`user-fetch`	`summarization`

Robots.txt control tokens such as Google-Extended and Applebot-Extended are not treated as distinct HTTP requester identities.

Common questions

Can a bot fake its User-Agent?

Yes. That is why the alpha labels such a request claimed. An origin-specific verifier can raise the status to verified or mismatch. WARD does not claim behavioural detection or universal crawler authentication.

Is this just robots.txt?

No. robots.txt is a discovery and exclusion convention. WARD evaluates a richer use policy inside the origin path and can change the delivered response. They can coexist, and future WARD adapters should consume established reservation formats.

Does a receipt prove training?

No. It proves the gateway's inputs and decision. It cannot see later storage, training, attribution, or use in another system.

Does WARD make a site EU AI Act compliant?

No single middleware can make that determination. WARD may help express, enforce, and document a policy, but applicability and compliance depend on legal and operational context.

Run the reference canary.

Compare human and declared-crawler responses on the live demo paths.

Live proof Integration guide