Google Has Already Named the AI Training Token for robots.txt

Today you write one rule per AI company and hope the next provider reads it the same way. Under the plan, a single policy token in robots.txt carries your training preference to every provider that joins the scheme. Google credits the AIPREF working group at the IETF for the wording.

The stated goal is one choice that covers many companies. That beats a file you edit every quarter as new crawlers appear. The announcement is short and worth reading in full: https://developers.google.com/search/blog/2026/10/ai-training-control

If your file opts out with the shared token and never mentions Google-Extended, Google plans to read that as a block on Google-Extended. One new line can then switch off training for Google as well. The person who wrote it may never notice.

Google-Extended is not a crawler in the ordinary sense. It has run since 2023 as a permission switch, and Googlebot still does the fetching. The token only tells Google what it may do with your text after the crawl: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Google ties a reward to one side of this choice. Sites that allow Google-Extended stay eligible for Gemini Apps answers, and those answers can send a reader back to the page. Allow the token and Google may also use your words to train future Gemini models. Block it and the answer placement goes with it.

This pair of lines blocks training for Google right now, in a file that every other crawler already reads:

User-agent: Google-Extended
Disallow: /

Swap that disallow line for Allow: / when you want to stay in Gemini Apps answers. Then hold the shared line until Google switches it on:

User-agent: __usage_ai-training
Disallow: /

The choice is per site, and often per section. A client case study and a pricing page rarely want the same answer, and a blog usually wants the opposite of a product page. Write the decision next to the rule so the next person doesn't have to guess. Google doesn't act on the shared line until early 2027, so adding it now costs nothing.

Google-Extended has no effect on whether a page appears in Google Search, and Google says it is not a ranking signal. Blocking training will not remove you from results. Allowing it will not buy you a better position.

Google also ties the plan to the EU Code of Practice for general purpose AI, which treats robots.txt as the main place for publishers to state machine-readable preferences. That makes the file a compliance artefact as well as a technical one. The ranking point still stands, and it matters when someone in the room says the change will hurt rankings.

We published a full guide to blocking AI bots in June 2026. Over the last 90 days it drew 269 impressions, 0 clicks and an average position of 70. The searches are real and the clicks are not. People who want this fixed read the documentation and get on with their day, so we wrote this post as a decision sheet rather than a sales page.

• Pick a policy per section, not per site, and write it down.
• Add the Google-Extended rule you want today, because that part is live.
• Diarise a review for the week Google switches on the shared token.

If you want a second pair of eyes on the file before the shared token lands, book a call: https://www.thesharpdigital.com/contact-us