Google and Microsoft back one robots.txt AI opt-out

What did Google and Microsoft announce?

Google and Microsoft announced they will respect a shared signal in robots.txt that says whether your content may be used to train AI models. The signal comes from the IETF's AI Preferences working group and is still a draft, not a finished standard.

Until now, you had to name each company's crawler and write separate rules. The Google Search Central announcement describes a single shared "policy token" that sets a broad preference across participating AI providers.

The IETF is the body that writes internet standards. Because this one is a draft, details could still change.

Is the shared robots.txt AI signal live yet?

No. Google and Microsoft have announced plans, but the shared robots.txt AI training signal is not active yet. Bing's new preferences take effect February 1, 2027, and Google plans to implement its support in 2027. Adding a line today will not change anything this week.

This is the part that is easy to miss. These are announcements of plans.

• Google: Search Engine Roundtable reports that Google plans to implement it in 2027.
• Microsoft: Bing's announcement says the new preferences take effect on February 1, 2027, as Microsoft crawls.

Think of adding the line as getting ready, not as flipping a switch.

Your old rules keep working. Google says its existing company-specific controls continue alongside the new signal.

What does Bing's split between training and grounding mean?

Bing now treats AI training and AI grounding as two separate decisions. Training means your content helps build or improve a model, while grounding means an AI answer draws on your pages and can cite them. Bing offers Training, Grounding and Limited Grounding settings, and you stay eligible for Bing search results either way.

According to Search Engine Roundtable's coverage, Microsoft is offering three settings:

• An AI Training setting, to allow or prevent use of your content for training.
• An AI Grounding setting, to allow or prevent use in generating AI answers.
• A Limited Grounding setting, a middle option: Microsoft's own AI products can still draw on your pages for answers, but outside services that rely on Microsoft's grounding cannot.

My own read is that blocking grounding probably means fewer chances to be cited in AI answers, so weigh that before you switch it off.

Per Bing's post, both settings default to Allow when you publish no preference. Training preferences can also be set page by page. Temporary grounding controls are planned in Bing Webmaster Tools for eligible sites that use NoArchive or NoCache, and Search Engine Roundtable lists September 30, 2027 as the date those temporary preferences expire.

Who does this affect?

The shared AI training signal affects every website with a robots.txt file, which is nearly everyone. It matters most to owners who publish original writing, photos, product descriptions or guides they would rather keep out of AI models. If you do not care, doing nothing is fine because the default is to allow.

Small shops with original product descriptions, photos or how-to guides have the most to think about. Whether to opt out is a business decision, not a technical one.

What should you do this week?

Open your robots.txt file this week, decide whether to opt out of AI training, and decide on Bing grounding separately. Keep your existing rules, and wait to paste in any new directive until you have read each provider's documentation. Then set a reminder for February 1, 2027.

1. Open your robots.txt. Type yourdomain.com/robots.txt into a browser. If your site builder hides the file, find the setting in its SEO section or ask your developer.
2. Decide your training stance. Opt out, or allow. This is a business call. My opinion: if your content is your product, opt out. If you want every possible mention, leave it open.
3. If you use Bing, decide on grounding separately. Blocking training doesn't require blocking AI answers. Weigh being cited against control.
4. Don't delete existing rules. Google says its company-specific controls continue alongside the new signal, so there is no reason to remove what you have.
5. Wait for the exact wording. Read each provider's documentation before you paste in a directive. The standard is still being finalized at the IETF, and syntax could change. Microsoft's own announcement is a Bing post by Krishna Madhavan.
6. Put a reminder in your calendar for February 1, 2027. That is when Bing's preferences take effect, and a good time to confirm your file says what you intend.

One caution. A robots.txt line is a request that participating companies have said they will honor. It is not a lock. It only covers providers that choose to participate.

Sources