Got a website? Then you’ve probably wondered whether AI companies are quietly feeding your pages into their next model. Until now, saying “no thanks” meant writing a separate rule for every AI crawler out there. That just got simpler. On October 30, Google, OpenAI, Microsoft and Anthropic backed one shared AI training opt-out signal for robots.txt.
It sounds like a tiny technical tweak. It kind of is. But it’s also the big four agreeing, for once, on a single way for you to say what they can and can’t do with your content.
So what actually changed?
The idea started as a draft proposal at the IETF, the group that writes a lot of the internet’s technical standards. Kevin Kelley of Anthropic wrote it. Then Google, OpenAI and Microsoft announced their support the same day the draft went public.
The proposal adds one new “policy token” to robots.txt. It’s called __usage_ai-training, and yes, those two underscores at the start are on purpose. If you add a rule for that token, you’re telling every participating AI company the same thing at once.
Before this, you had to name each company’s crawler. GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. Miss one, and that company could still train on your pages. So one shared line saves a lot of guesswork.
Google says the new control fits the EU’s Code of Practice for general-purpose AI. That code “recognizes robots.txt as a primary method for web publishers to express machine-readable preferences.” In other words, Europe already treats this file as the place where your wishes live.
How the AI training opt-out works, in plain English
First, a quick refresher. Robots.txt is a small text file that sits at the root of your website. Crawlers read it before they visit your pages, and it tells them where they’re welcome.
The new AI training opt-out adds a group for the shared token. Here’s the example from the draft, which blocks AI training across a whole site except its press kit:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That’s it. Three lines. Your other robots.txt rules stay exactly as they are.
There’s one neat detail on Google’s side. If you use the shared token but don’t mention Google-Extended, Google plans to treat that as if you’d blocked Google-Extended too. So you don’t need to double up. Also, Google, OpenAI and Anthropic say their older, company-specific controls will keep working alongside the new one.
Who’s doing what, and when
Here’s the catch: nobody’s flipping the switch on the same day.
The early movers
Anthropic proposed the signal, and according to Coywolf it’s the only company using it today. Kelley still admitted, “It’ll take us some time to add support across our services.” OpenAI says it’ll implement the signal “in the coming months” and that it “will honor the voluntary signal.”
The 2027 crowd
Microsoft will start honoring these preferences on February 1, 2027. It’s also adding a separate setting for grounding, which is when an AI answer pulls from a live web page. Microsoft’s Krishna Madhavan put it well: “Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”
Google’s timeline is the vaguest. It aims to support the control “in 2027,” and its crawler docs point to early that year. Meanwhile, Meta’s communications head Andy Stone said on X that he backs a common signal. But he didn’t say whether Meta will actually honor it.
The fine print nobody puts in the headline
Okay, time for some honesty. This is a polite request, not a lock.
The token expresses a preference about how your content gets used. It isn’t an order, and nothing forces anyone to obey it. Coywolf put it bluntly: “the signal isn’t enforceable.” If a crawler ignores robots.txt, you’d need bot-blocking tools, like Cloudflare’s, to actually stop it.
Also, private deals win. If a publisher signs an agreement with an AI company, that contract can override the robots.txt preference. And Microsoft says the preferences “do not establish licensing or payment arrangements.” So this won’t put any money in your pocket.
Then there’s the old stuff. Nothing in these announcements says the signal pulls your pages out of models that already trained on them. It’s about what happens next, not what happened before.
Finally, it’s still a draft. The IETF hasn’t endorsed it, and the details could change. Honestly, that’s fine. Standards usually start messy.
Okay, but should you use the AI training opt-out?
It depends on what you want from AI. If your content is your product, like news or research, blocking training probably makes sense. You’d keep a bit more control over your work while the bigger fights over payment play out.
But if you’re a startup, a nonprofit or a small business, think twice. In our view, AI assistants are turning into a front door to the web for a lot of people. Microsoft warns that blocking grounding “will significantly reduce your content’s visibility in AI-generated experiences.” At Microsoft, training and grounding are separate settings, though, so read each one carefully before you copy and paste anything.
We’ve written before about AI agents poking around Wikipedia and how cheaply AI agents can hack company websites. The common thread? AI is already out there reading the web, sometimes in ways site owners never planned for. A shared rulebook won’t fix all of that. Still, it’s a start.
So here’s our take. Spend ten minutes this week looking at your robots.txt. Decide what you’re comfortable sharing. Then watch whether the AI training opt-out actually gets respected once the 2027 deadlines hit. That’s the real test.
Sources: Google Search Central, Coywolf, The Next Web and the IETF draft.