Google’s Shared AI Training Control: How __usage_ai-training Works
What Google’s October 2026 announcement means for publishers, how the proposed policy token works, and what to do before support rolls out in 2027.
Website owners have spent the last few years managing AI-training preferences vendor by vendor: one rule for Google, another for a different AI provider, and often a separate crawler policy on top of that.
On October 30, 2026, Google announced that it plans to support a shared AI-training preference in robots.txt. The proposal is based on work around the IETF AIPREF effort and introduces a cross-vendor policy token:
User-Agent: __usage_ai-training
Disallow: /The idea is simple: instead of repeating a different training opt-out for every participating provider, a site could express one broad preference and then override it for individual providers when needed.
But there is an important qualification: this is not an active universal standard today. Google says it intends to support the shared signal in 2027, while the underlying document is still an Internet-Draft and explicitly remains work in progress.
That means publishers should understand the mechanism now, prepare their policy, and avoid treating the token as a magic switch that every crawler already obeys.
What Google Announced on October 30, 2026
Google Search Central announced plans to support a shared, machine-readable AI-training control in robots.txt. The goal is to let website operators express training preferences across multiple AI providers through a single policy token rather than maintaining a growing collection of provider-specific controls.
Google says the proposal builds on work from the AIPREF community at the Internet Engineering Task Force and complements its existing Google-Extended control.
Official source: Google Search Central: Supporting a shared AI training control for websites.
The proposed token is:
User-Agent: __usage_ai-trainingDespite the familiar User-Agent syntax, this should not be treated like a normal crawler identity. In the draft, it acts as a policy token that communicates a preference about how retrieved content may be used.
Google plans support for 2027, not immediate enforcement
Google's current crawler documentation says it intends to support the AIPREF shared machine-readable training preference signal in early 2027.
Until support actually rolls out, publishing the token does not mean every AI company—or even every Google system—will interpret it.
This timing distinction matters. A tutorial that says “add this line today and all AI training stops” would overstate what has actually been announced.
How the Proposed Policy Token Works
The October 2026 Internet-Draft defines the policy token __usage_ai-training. It deliberately reuses ordinary robots.txt group structure and path matching instead of inventing a new file format.
A simple site-wide training opt-out would look like this:
User-Agent: __usage_ai-training
Disallow: /Under the proposed semantics, this expresses a preference against the use covered by the AI-training policy for the entire site.
A site can also express a path-specific preference:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example says: do not use the site's content for the covered AI-training purpose, except for content under /press-kit/.
The draft keeps the normal robots.txt path model, including ordinary group structure and longest-match behavior.
Draft source: Machine-Readable Training Preference Signal.
This is still an Internet-Draft
The document published on October 30, 2026 is an Internet-Draft. The IETF explicitly describes Internet-Drafts as working documents that may be updated, replaced, or obsoleted.
So treat the exact token and interpretation as a current proposal, not as an immutable final RFC.
AI-Training Preference Is Not the Same as Crawl Blocking
This is the most important implementation detail.
The proposed AI-training policy is independent from crawler access rules.
For example, this:
User-Agent: __usage_ai-training
Disallow: /expresses a preference about AI-training use. It does not automatically mean:
User-Agent: Googlebot
Disallow: /Those are different decisions.
The draft explicitly says content-usage preferences and traffic-management signals can coexist in robots.txt while remaining separate concepts.
That distinction is particularly important for publishers that want to remain visible in search engines while declining certain training uses.
Do not block Googlebot just to opt out of AI training
If a page is intended to rank in Google Search, blocking Googlebot is generally the wrong mechanism for expressing an AI-training preference.
Googlebot rules affect Google Search crawling. The shared AI-training token is intended to communicate a usage preference instead.
Rubic8's existing guide to Cloudflare AI crawler controls discusses the same broader principle: search discovery, AI training, and agent access are related operational questions, but they are not interchangeable.
How This Interacts With Google-Extended
Google already provides Google-Extended as a robots.txt product token.
Google says Google-Extended controls whether content that Google crawls can be used for specified Gemini training and grounding uses. It also says Google-Extended does not affect inclusion or ranking in Google Search.
For example:
User-Agent: Google-Extended
Disallow: /remains a Google-specific control.
Google says existing company-specific controls will continue to work alongside the shared signal.
What if both rules are present?
Google's announcement gives an important example of precedence.
If a website uses the shared policy token to opt out of training and does not explicitly specify Google-Extended, Google plans to interpret the shared opt-out as though Google-Extended were disallowed.
But company-specific rules can still be used for more targeted choices.
A configuration might eventually look like this:
User-Agent: __usage_ai-training
Disallow: /
User-Agent: Google-Extended
Allow: /The broad rule expresses a general preference against AI training, while the Google-specific rule expresses a different choice for Google.
Because implementation details may evolve before 2027 support is live, publishers should re-check Google's documentation before deploying this pattern in production.
Will the New Signal Affect Google Search Rankings?
Google has not announced the shared AI-training preference as a ranking signal.
More importantly, Google explicitly states that Google-Extended does not affect a site's inclusion or ranking in Google Search.
The proposed shared policy is about content use, not a new SEO optimization technique.
That means publishers should avoid claims such as:
- “Adding the token will improve rankings.”
- “Opting in will make Google AI Overviews cite your site.”
- “Opting out will remove pages from Google Search.”
None of those conclusions follows from the current documentation.
If your objective is visibility in Google AI Mode or AI Overviews, use Google's Search-specific guidance instead. Rubic8's Google AI Mode publisher guide covers that separate search-discovery workflow.
Why User-Agent: * Does Not Replace the Policy Token
A common mistake will be assuming this existing rule:
User-Agent: *
Disallow: /private/automatically expresses an AI-training preference.
Under the current draft, it does not.
The draft specifically says wildcard groups such as User-Agent: * must not be interpreted as matching the __usage_ai-training policy token.
That design helps keep ordinary crawler traffic management separate from content-use preferences.
What Happens to Crawlers That Do Not Understand the Token?
Legacy robots.txt parsers should continue behaving normally.
A crawler or parser that does not implement the new policy token will generally treat the unfamiliar group as irrelevant to its own crawler identity.
This backward-compatible behavior is one reason the proposal uses familiar robots.txt syntax rather than a completely new protocol.
But backward compatibility should not be confused with universal enforcement. An unknown token being harmless to older parsers does not mean those parsers respect the associated AI-training preference.
A Practical Publisher Decision Framework
| Publisher objective | Relevant control | Do not confuse it with |
|---|---|---|
| Stay discoverable in Google Search | Googlebot access, indexability, canonical and search controls | AI-training permission |
| Decline broad AI-training use | Shared __usage_ai-training signal once supported | Blocking all crawler traffic |
| Set a Google-specific Gemini preference | Google-Extended | Google Search ranking |
| Prevent a specific crawler from fetching content | That crawler's robots.txt group or infrastructure controls | A content-use policy token |
| Keep a page out of Google Search | noindex where appropriate | AI-training opt-out |
What Publishers Should Do Now
The strongest action today is not blindly adding a new line to robots.txt. It is preparing a policy and deployment process for 2027.
1. Audit your existing robots.txt
Identify current groups for Googlebot, Google-Extended, other AI-specific crawlers, security-generated rules, sitemaps and custom directives.
You can use Rubic8's Robots.txt Generator & Tester to inspect and stage rules without editing production first.
2. Write down your content-use policy
Decide whether your preference should be site-wide, path-specific, different for press or marketing assets, or different for particular AI providers.
3. Keep search crawling and AI-training preferences separate
Do not block broad search crawlers unless blocking crawling is genuinely your objective. Test search-critical paths after every robots.txt change.
4. Watch the draft and vendor documentation
Before enabling the new policy token in production, confirm the current token name, which providers support it, how precedence works, and whether your CDN or bot-management layer rewrites robots.txt.
5. Record the reason for every rule
For each policy group, keep a short change note such as: “This rule expresses our preference against generative-AI model training. It is not intended to block search crawling.”
A Safe Staging Example
Suppose a publisher eventually wants to decline general AI-training use while keeping a press-kit directory available for that purpose.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Before deployment:
- Confirm that the providers you care about support the signal.
- Confirm that the final specification still uses the same token and semantics.
- Review existing provider-specific groups such as Google-Extended.
- Verify that Googlebot and other search crawler rules remain unchanged.
- Test critical public URLs after publishing the new robots.txt file.
- Monitor crawl and search performance for unrelated regressions.
Should Rubic8 Add This to Its Robots.txt Tool?
Yes—but the best implementation should reflect the technology's current status.
Rubic8's Robots.txt Generator already supports custom directives, so advanced users can stage the proposed group today.
A future dedicated preset could offer a shared AI-training preference selector, site-wide or path-specific Allow/Disallow rules, a warning that the signal is about usage rather than crawling, Google-Extended override examples, and a status note showing which major providers currently support the proposal.
That feature should not be marketed as universal enforcement until real provider support is confirmed.
What Not to Do
- Do not claim the token works universally today. Google support is planned for 2027.
- Do not describe the Internet-Draft as a finished IETF standard.
- Do not replace Googlebot rules with the AI-training policy token.
- Do not assume
User-Agent: *expresses the same preference. - Do not promise SEO gains, AI citations, or traffic from opting in.
- Do not assume opting out removes pages from Google Search.
- Do not delete provider-specific controls until you understand their precedence.
Frequently Asked Questions
What is __usage_ai-training?
It is the policy token defined in the October 2026 Internet-Draft for expressing a machine-readable preference about generative-AI model training through robots.txt.
Does Google support it today?
Google has announced plans to support the signal in 2027, and its crawler documentation currently says early 2027. Publishers should not assume it is already active.
Does it block crawlers?
Not by definition. The proposed token expresses a content-use preference. Ordinary crawler access rules remain separate.
Will it affect Google Search rankings?
Google has not presented the shared training preference as a ranking signal. Google also states that its existing Google-Extended control does not affect inclusion or ranking in Google Search.
Can I opt out site-wide but allow one directory?
Yes, the current draft supports normal robots.txt path matching, so a broad Disallow can be combined with a more specific Allow such as /press-kit/.
Should I remove Google-Extended when this launches?
Not automatically. Google says company-specific controls will continue to work alongside the shared signal. Review the final documentation and decide whether you need a provider-specific override.
Final Takeaway
Google's October 30 announcement is important because it points toward a simpler model for publisher control: one shared robots.txt signal for AI-training preferences, with provider-specific rules available when publishers need finer control.
But the practical message for November 2026 is prepare, do not overclaim.
The proposed __usage_ai-training token is still tied to an Internet-Draft, and Google support is planned for 2027. Use the time before rollout to audit your robots.txt file, separate search crawling from content-use preferences, document your actual policy, and build a deployment process that can be tested safely.
If your goal is to remain discoverable in Search while declining certain AI-training uses, the most important technical habit is simple: treat crawling, indexing, AI training, and AI-grounding controls as separate decisions.
Official Sources
- Google Search Central: Supporting a shared AI training control for websites
- Google Crawling Infrastructure: Common crawlers and Google-Extended
- IETF Datatracker: Machine-Readable Training Preference Signal Internet-Draft
Rubic8 Editorial Team
Editorial Team
Rubic8 creates practical guides and free tools for developers, webmasters, and digital publishers. Our fast-changing technical content is reviewed against current primary documentation before publication.
