OpenAI, Google, Microsoft and Anthropic back new opt-out standard for AI scraping
Four leading AI companies say they will adopt a shared standard that lets publishers opt out of AI scraping across multiple companies at once, using a signal placed in robots.txt files.
OpenAI, Google, Microsoft and Anthropic have said they will adopt a new standard that allows website publishers to opt out of being scraped by artificial intelligence systems. Rather than setting training preferences separately with each company, publishers would be able to apply one set of choices across all of them.
The shared signal, known as a "policy token," was proposed by Kevin Kelley, a technical lead for standards at Anthropic. The proposal is being developed within the Internet Engineering Task Force, the long-established body that shapes many of the technical standards underlying the internet. The signal would live inside a website's robots.txt file, the plain-text document that has for years told web crawlers which parts of a site they may or may not access.
The robots.txt file has traditionally governed a range of crawler activity, including search indexing and archiving. Its role has become more contested as AI companies collect large volumes of web content to train their models. Publishers have increasingly sought ways to control whether their work feeds that process, and until now they have generally had to manage those preferences company by company, a process that is slow and difficult to keep consistent across a large number of AI developers.
By placing the control in a single, widely recognized location, the proposed standard would let publishers express their AI training preferences once and have them respected by participating firms. The approach builds on a convention that crawler operators have long been accustomed to honoring, which is part of why its backers see it as a practical route rather than a wholly new mechanism. The move also reflects a broader effort among AI developers to respond to concerns from content creators about how their material is gathered and used.
The development was first reported by Search Engine Roundtable before being covered in more detail by The Verge. The available reporting does not specify how the policy token would be formatted, how compliance would be verified, or when the standard might be finalized and rolled out. Those open questions will likely determine how much practical weight the commitment carries once it moves beyond the proposal stage.
For publishers, the significance lies in consolidation. A single signal recognized by the largest AI developers would reduce the administrative burden of managing separate settings across each company's crawler, and it would give content owners one clear place to state their preferences. Whether the standard becomes a dependable tool will depend on its adoption by additional companies and on the technical details still being worked out within the standards body.
5
