A Shared Robots.txt Token
Google, Microsoft, and OpenAI have unveiled intentions to adopt a mutual robots.txt line pertaining to AI training preferences.
This initiative originates from an Internet Draft submitted by Kevin Kelley of Anthropic to the AI Preferences working group within the IETF.
The draft utilizes a conventional robots.txt rule group, allowing a site to stipulate a policy token labeled __usage_ai-training beneath a User-Agent line.
An illustrative example demonstrates User-Agent: __usage_ai-training Disallow: / and Allow: /press-kit/.
The document categorizes AI Training as the utilization of content to amend the learning parameters of a generative AI model.
Notably, models executing exclusively non-generative functions, such as classification or ranking, are exempt from this definition.
Furthermore, the draft articulates that a wildcard group, exemplified by User-Agent: *, cannot correspond to the token.
It asserts that the absence of a __usage_ai-training group in a file signals a lack of preference, a truth applicable to all currently published robots.txt files, as stated by Kelley.
Additionally, the draft clarifies that this token does not govern crawling activities, portraying the mechanism as a preference signal rather than a mandate altering the nature of the data a crawler retrieves.
Google and Microsoft Split Roles
In a blog post by John Mueller of Google, the interpretation of a site opting out of training using the shared line—without specifically mentioning Google-Extended—is that Google will consider it as a disallowing of Google-Extended usage.
According to Google’s crawler documentation, the company aims to implement this control by 2027. This documentation also clarifies that Google-Extended dictates whether content may contribute towards training forthcoming Gemini models and its applicability for grounding within Gemini Apps and on Vertex AI.
In parallel, Bing’s principal product manager, Krishna Madhavan, posted that Microsoft will adopt the shared line for training, while introducing a distinct AI Grounding preference that is applicable solely at the page level.
Microsoft has announced that it will commence honoring these new options on February 1, 2027, discontinuing recognition of previous tags for training or grounding after March 1, 2027.
The company has indicated that either of these restrictions will curtail a brand’s visibility within AI-generated responses but will not impede its presence in Bing search results.
Implementation Timelines and Limits
PPC Land reported that Kevin Kelley of Anthropic informed an IETF mailing list on October 30, 2026, that his firm has already initiated support for the draft.
Microsoft plans to commence this support on February 1, 2027, while Google is preparing for an early 2027 launch. OpenAI has indicated its intention to implement this signal in the forthcoming months.
It is crucial to note that the draft itself is classified as an individual Internet-Draft and does not signify formal IETF standardization activities.
The Search Engine Journal’s analysis reveals that, according to set timelines, none of these three corporations presently support the line, and both Google and Microsoft have articulated their distinct approaches to handling it.
The Next Web detailed that Microsoft will begin recognizing these new preferences as of February 1, 2027, replacing the NoArchive and NoCache tags that have been in use since 2023.
Furthermore, The Verge noted that Andy Stone, Meta’s communications head, expressed strong support for a unified signal, stating, “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” although he refrained from announcing whether Meta would adhere to these new standards.
Source link: Newscord.org.
