Your opt-out is now law. A line in robots.txt isn't proof you used it

Slava Spitsyn · June 19, 2026 · 3 min read

The EU now recognizes a machine-readable "do not train on this" as a legal reservation of rights. Naturally, the first thing everyone reaches for is robots.txt — the file the web has used to talk to crawlers for thirty years. It feels like the obvious tool. It's also the wrong one for the job the law just created.

What's new

• Under the DSM Directive and the EU AI Act, a rightsholder can reserve their work from AI training through machine-readable means — and GPAI providers are expected to respect it.
• This has pushed robots.txt and newer signals (like dedicated AI-crawler directives) into the spotlight as the everyday way to "opt out."
• The assumption baked into all of it: if I declare it in a file, I'm protected.

How it works

robots.txt does one thing, and does it by design as a request:

• It sits at the root of your site and asks well-behaved crawlers to stay out of certain paths.
• Compliance is voluntary. There is no handshake, no receipt, no enforcement built in.
• It records nothing. The file states your wish; it never logs who read it, who ignored it, or what they took on the way past.

It was built for an era when breaking the rule wasn't worth much. That era is over.

Behind the news

The law changed the stakes, but not the tool. And the tool has two gaps that matter now:

• It can't prove your declaration existed at the moment of the crawl. A reservation only protects you if you can show it was in place when your work was taken. A static file with no timestamped record of access leaves you arguing about the past with nothing in hand.
• It can't show who ignored it. The crawlers that respect robots.txt were never the threat. The ones that matter walk straight past it — and leave the same anonymous trace as everyone else.

So you can do everything "right," reserve your rights exactly as the law intends, and still have no way to demonstrate it when it counts.

A "Do Not Enter" sign taped to an unlocked door is a statement of intent. It is not a record of who walked in.

Why it matters

The reservation and the proof are two different things, and only one of them is in the file. The EU gave the reservation legal weight. What turns that weight into something you can stand on is the record: who accessed your content, when, and whether your declaration was in force at that moment.

Without the record, a perfect opt-out is still just a request. With it, the same opt-out becomes evidence.

How I see it

I don't think robots.txt is the villain — it's a thirty-year-old request mechanism being asked to do a job it was never designed for. The job the law actually created has two parts, and the file only does the first:

• Declare how your content may be used — machine-readably, the way the EU recognizes.
• Record who came and what they took, so the declaration is provable, not just present.

That second part is what I'm building with WARD. robots.txt asked nicely. The record is what holds up.

→ getward.org