One Token for All: The New AI Training Control in robots.txt
By Arash Latifi — Founder, The DevArt
Google, OpenAI, Microsoft, and Anthropic announced a shared robots.txt token called __usage_ai-training. What it actually does, who honors it today, and what site owners should do right now.
TL;DR — Google, OpenAI, Microsoft, and Anthropic jointly announced a shared robots.txt token,
__usage_ai-training, so site owners can set one AI-training preference for all of them. Catch: only Anthropic honors it today; Google plans early 2027.
If you manage a website, you know the annoyance: a separate robots.txt rule for every AI crawler — GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and Microsoft never had a clean "training only" block at all. The October 30 announcement fixes exactly that mess.
Where this came from
The AIPREF working group at the IETF has been working on a web standard for machine-readable content preferences against AI. Google's Search Central post announced support: instead of per-company instructions, you write one shared "policy token" in robots.txt, and participating AI providers honor it.
Important framing: this is a voluntary signal, not a law. No crawler that wants to misbehave is obligated to obey it. (I've covered the brand/reputation side of AI answers before in Google AI answers and your reputation; this post is the technical companion.)
What the rule looks like
Block AI training site-wide while keeping your press kit open:
One User-Agent, one outcome across four major providers.
Google's subtle rule: if you opt out via the token without explicitly specifying Google-Extended, Google will interpret it as if you had disallowed Google-Extended. That removes your site from grounding in Gemini Apps — and the clicks that come from Gemini answers. It's a conscious trade between "don't train on my content" and "don't lose AI-sourced traffic."
Microsoft also introduced DisallowExtended, which applies only to its own experiences like Copilot and 365 — a non-standard move Coywolf compared to the old Internet Explorer play.
The uncomfortable reality: who honors it today
| Provider | __usage_ai-training support |
| --- | --- |
| Anthropic | Live now |
| OpenAI | Coming months |
| Google | Early 2027 |
| Microsoft | February 2027 |
So if you update robots.txt today, only Anthropic actually honors it. The legacy tokens (GPTBot, Google-Extended, ClaudeBot) still work and should stay alongside the new one for now.
Checklist: what to do now
- [ ] Open your current robots.txt and see which AI bots you block or allow today
- [ ] Make the strategic call: should your content train models or not? (business decision, not technical)
- [ ] If "no" — add the
__usage_ai-trainingblock, but keep existing tokens - [ ] If "yes, but I want Gemini traffic" — explicitly allow
Google-Extended - [ ] Remember robots.txt doesn't stop bad-faith bots; you need WAF/rate limiting (e.g., Cloudflare) for those
- [ ] Calendar a re-check for early 2027 when Google's support lands
What this means for small shops
If you run an e-commerce site with buying-guide content (size guides, comparisons, FAQs), structured data and depth still matter for search and AI answers — blocking training does not block display. Don't conflate "don't train on me" with "don't show me."
But if your business counts on referral traffic from Gemini or ChatGPT answers, withholding training permission can reduce your presence in those answers over time. Per Google's announcement, explicitly allowing Google-Extended is what keeps you eligible for clicks via grounding in Gemini Apps.
FAQ
Q: Can I drop GPTBot and the old tokens now?
Not yet. Anthropic is the only one honoring the shared token today; OpenAI follows in the coming months, Google in early 2027, Microsoft in February 2027. Keep the old tokens until then.
Q: Does this remove my site from search results?
No. The signal controls model training only, not search indexing. As long as you don't break robots.txt for regular crawlers, your rankings are untouched.
Q: What about crawlers that ignore it?
Nothing happens to them. robots.txt is voluntary; stopping bad-faith bots requires other layers like a WAF, rate limiting, or bot-management services.