Your opt-out is now law. A line in robots.txt isn't proof you used it

Slava Spitsyn · June 19, 2026 · 3 min read

The EU now recognizes a machine-readable "do not train on this" as a legal reservation of rights. Naturally, the first thing everyone reaches for is robots.txt — the file the web has used to talk to crawlers for thirty years. It feels like the obvious tool. It's also the wrong one for the job the law just created.

What's new

• Under the DSM Directive and the EU AI Act, a rightsholder can reserve their work from AI training through machine-readable means — and GPAI providers are expected to respect it.
• This has pushed robots.txt and newer signals (like dedicated AI-crawler directives) into the spotlight as the everyday way to "opt out."
• The assumption baked into all of it: if I declare it in a file, I'm protected.

How it works

robots.txt does one thing, and does it by design as a request:

• It sits at the root of your site and asks well-behaved crawlers to stay out of certain paths.
• Compliance happens on the crawler side. There is no handshake, receipt, or origin enforcement built into the file format.
• It records nothing. The file states a rule; it does not log which requester read it or what response the origin served.

It was built for an era when breaking the rule wasn't worth much. That era is over.

Behind the news

The law changed the stakes, but not the tool. And the tool has two gaps that matter now:

• It does not itself timestamp which version existed for a request. That evidence has to come from deployment logs or another record.
• It can't show who ignored it. The crawlers that respect robots.txt were never the threat. The ones that matter walk straight past it — and leave the same anonymous trace as everyone else.

So you can do everything "right," reserve your rights exactly as the law intends, and still have no way to demonstrate it when it counts.

A "Do Not Enter" sign taped to an unlocked door is a statement of intent. It is not a record of who walked in.

Why it matters

The reservation and the origin record are two different things, and only one of them is in the file. A useful record can show the declared or verified requester, which policy version was in force, and what the origin decided. It still cannot prove downstream training.

Without the record, a perfect opt-out is still just a request. With it, the same opt-out becomes evidence.

How I see it

I don't think robots.txt is the villain — it's a thirty-year-old request mechanism being asked to do a job it was never designed for. The job the law actually created has two parts, and the file only does the first:

• Declare how your content may be used — machine-readably, the way the EU recognizes.
• Record the requester confidence, policy version, and response decision.

That second part is what I'm building with WARD. robots.txt asked nicely. The record is what holds up.

→ getward.org