Blog ·
AI training control in robots.txt: what owners should do
You can decide whether AI companies may use your website for AI training, and Google just said it will make that choice simpler. On October 30, 2026, Google announced plans to support a shared control in robots.txt that works across AI providers. Nothing changes for your rankings today, and the new signal is not live yet. The only job for this week is to look at your robots.txt file and decide what you want it to say.
What Google announced on October 30
John Mueller wrote that the control is based on work from the IETF's AIPREF working group. Google says it will help websites express their AI training preferences across multiple AI providers in a single step. A site would use one shared "policy token" in robots.txt instead of naming each provider.
Google says it aims to support the control in 2027. Until then, the controls you have today still work. Google also says that if the token opts out of training and you have not named Google-Extended, Google plans to treat it as if you had disallowed Google-Extended. Company-specific controls keep working next to the shared one.
The signal is still a proposal. The draft behind it is not endorsed by the IETF and has no formal standing in its standards process, and it expires on May 3, 2027 unless it is renewed. Search Engine Roundtable quoted Nate Hake, who says he takes part in the IETF group, saying "it's not actually done at the ietf level -- and some significant areas of dispute remain". Treat the details as likely to change.
What this control does not do
It is a preference about training. It does not remove your site from Google. Google's crawler page says "Google-Extended does not impact a site's inclusion in Google Search", and that page says it is not a ranking signal either. Your Google Business Profile is a separate system and this change does not touch it.
The draft is also clear that the token is a signal, not a lock. It states that "This policy token is not a directive for crawling behavior", and it allows that private agreements may override a preference. It also says a wildcard group such as User-Agent: * does not match the token. A blanket "block everyone" rule will not opt you out of training by itself.
Google's own robots.txt guide adds a second warning. It says robots.txt "is not a mechanism for keeping a web page out of Google". If a page must stay private, such as a customer portal, do not rely on this file. Use a password.
Should you opt out of AI training?
Training and appearing in AI answers are not the same choice. OpenAI's bot page says GPTBot indicates a site's content should not be used in training generative AI foundation models when blocked. The same page says "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers". It also says each setting is independent of the others. You can block one and allow the other.
Google draws a similar line. Sites that allow Google-Extended are, in Google's words, eligible to appear in responses and receive clicks via grounding in Gemini Apps. So opting out of training may affect whether your site is used in Gemini Apps answers, per Google. Think before you block everything. Our guide to AI search for local service businesses covers how to show up there.
Here is a plain way to decide. Neither answer is wrong. The point is that you choose on purpose.
- Allow training if you want the widest reach and your pages hold only public facts: services, towns, hours, phone number.
- Opt out of training if your site holds original photos, written guides, or price sheets you do not want used to build someone else's product.
- Keep search and assistant crawlers allowed in either case if you want to be found by customers who ask AI tools for a local provider.
Check your robots.txt file this week
You do not need a developer for the first look. You only need your web address and ten minutes.
- Type yourdomain.com/robots.txt into your phone's browser. Take a screenshot.
- If you get a not found page, you have no file. That is fine. It means every crawler may visit.
- Look for the lines Disallow: / next to User-agent: *. That blocks all crawlers, including the ones that send you customers. Fix it first.
- Look for any rule that names GPTBot, Google-Extended or other AI crawlers. Write down who set it and why.
- Ask whoever built your site, or your host, who can edit the file. Get a name and a phone number.
A blanket block is the costly mistake. It can happen that a rebuild carries over a staging rule that tells every crawler to stay away. If your site is hard to find, start with our checklist on why your website is not showing up on Google before you worry about AI training.
What to write now and what to wait on
Use the controls that exist today. For Google, that is the Google-Extended token. For OpenAI, it is GPTBot. Each company sets its own names, so check its own page before you add a line. Write the rule that matches your decision above, and leave search crawlers such as OAI-SearchBot and Googlebot alone if you want to be found.
Wait on the shared token. The draft names it __usage_ai-training, but Google's support is planned for 2027 and the draft may change. Search Engine Roundtable notes Google's plan is similar to announcements from Bing, OpenAI and Anthropic. Put a calendar reminder on your phone for early 2027 to check Google's updates.
Do not paste a rule you found in a forum into a live file. A mistake in robots.txt can hide pages from crawlers. Ask your site builder to test the change, then reload yourdomain.com/robots.txt and read it back.
Keep the site that earns your calls working
This is a small decision next to a site that loads slowly, shows the wrong phone number, or sits behind a builder you cannot log into. Google's own robots.txt page says the file is mostly about managing crawl traffic. Your main job is a site a customer can read and call from. Our piece on the Gemini small business tools and your Business Profile shows where your profile and site meet. You can also read what owners did in our case studies.
- Open yourdomain.com/robots.txt today and screenshot it.
- Confirm nothing blocks all crawlers.
- Decide: allow training or opt out. Write the decision down.
- Ask who can edit the file, and have them test any change.
- Set a reminder to revisit when Google ships support in 2027.
If you want this done from the site you already have, Vortex Personal is $59 a month for one live site. Pro is $449 a month for up to 10. Business is $2,499 a month for up to 150. Hosting is included. A year is ten times the month price. The old site stays up until you switch.
If you cannot get into your old site to change any of this, start a rebuild from the URL you already have.
Sources
Google Search Central Blog: Supporting a shared AI training control for websites (October 30, 2026)
Google Search Central: Google's common crawlers
Google Search Central: Introduction to robots.txt
IETF Datatracker: draft-kelley-aipref-training-preference-signal-00 (October 30, 2026)
OpenAI: Overview of OpenAI crawlers
Search Engine Roundtable: Google Adopts New AI Training Control (October 30, 2026)
