One robots.txt line is meant to shield websites from training by four big AI developers

Website owners will soon no longer have to block every AI crawler one by one. OpenAI, Google, Microsoft and Anthropic say they will honour a common signal that lets a site state in its robots.txt file that its content may not be used for AI training. The Verge reports, following an initial story by Search Engine Roundtable, that this is a so-called policy token proposed within the standards body IETF by Kevin Kelley, a technical lead for standards at Anthropic. The proposal builds on the work of the IETF's AIPREF working group.

According to Search Engine Roundtable, Google has already updated its documentation: the company intends to support the signal in early 2027 and will treat an opt-out via the shared token as if the site had excluded its own Google-Extended crawler setting. Company-specific controls will continue to exist alongside it. Microsoft goes further and gives publishers separate choices for training and for grounding, the use of content to back up AI answers. A site that refuses training can therefore still be cited by Copilot, and regular Bing rankings are unaffected. Microsoft plans to start applying the new preferences from 1 February 2027.

Anthropic already uses the signal and OpenAI promises it within the coming months, The Verge writes. Meta also said it supports a common signal but did not say whether it will follow it. The initiative comes as publishers grapple with a flood of AI crawlers. Earlier attempts to contain them include the RSL licensing standard, which lets sites charge crawlers, and Cloudflare's decision to block AI crawlers by default. Transparency: this edition was compiled by Claude, made by Anthropic.