AI providers back a shared training signal, but rollout is still ahead

On 30 October, OpenAI and Google each said they intend to support a shared robots.txt preference for AI training, based on work being discussed through the IETF’s AI Preferences group. The announcements describe a future control, not a feature website owners can assume is already active: OpenAI says implementation details will come later, while Google places its target in 2027.

A proposal and adoption plans, not a finished standard

The IETF Datatracker currently labels the cited text an Internet-Draft. It is a work in progress that can change, rather than a published RFC or a final implementation contract. The distinction matters because a draft can explain the direction of the discussion without settling the behavior of every crawler, hosting platform or content-management system.

OpenAI says it plans to honor the shared signal in the coming months and that its current crawler controls continue to apply. Google says it plans to support the mechanism in 2027 and that its existing Google-Extended controls remain alongside the proposed shared preference. These are provider statements about intended support; they are not evidence that all AI services have adopted the same interpretation.

Training preferences are not search indexing controls

The new signal discussed in the announcements concerns use of web content for AI model training. It should not be presented as a universal switch for ordinary search crawling, indexing, search ranking, or every way an AI product may retrieve web pages. Google’s post distinguishes the training preference from Google-Extended and says current provider-specific controls continue. Site teams should preserve those controls until each provider publishes concrete instructions for the shared mechanism.

That distinction also limits what a robots.txt entry can promise. The announcements describe a machine-readable preference that participating providers say they will honor. A file does not technically prevent an uncooperative crawler from requesting a public page, and the announcement does not resolve legal questions about content use.

What site operators can do now

Website owners can inventory the provider-specific crawler rules they already use, record the purpose of each rule, and identify who can update robots.txt. They can also wait for the final draft state and provider implementation documents before changing production rules. When support arrives, operators should test the exact syntax and expected behavior against authoritative provider documentation, check that existing search and user access remain intact, and retain a record of the change.

The reader-facing value of this announcement is its timing boundary: there is a shared direction, but no reason to treat the proposal as live everywhere today. BijliCoin found no existing canonical page that explains this particular training-versus-search distinction; its published agent source-verification workflow covers a different reader task, so this story does not link to it.

Primary sources: OpenAI’s announcement, Google Search Central’s announcement, and the IETF Datatracker draft record.

Illustration: AI-generated conceptual art.