comp-journalism EN

Anthropic, OpenAI, Google, and Microsoft will make it easier for publishers to opt out of being scraped

[AI & Journalism

AI](https://www.niemanlab.org/beats/ai-journalism/) [Search & Platforms

Search & Platforms](https://www.niemanlab.org/beats/search-platforms/) [Audience & Creators

Audience & Creators](https://www.niemanlab.org/beats/audience-creators/) [Local News

Local News](https://www.niemanlab.org/beats/local-news/) [Reporting & Production

Reporting](https://www.niemanlab.org/beats/reporting-production/) [Business & Revenue

Revenue](https://www.niemanlab.org/beats/business-revenue/)

About Nieman Lab Subscribe Archives Contact

Nieman Foundation Nieman Fellowships Nieman Reports Nieman Storyboard

Appearance

Auto

Light

Dark

Text size

A

A

A

Get Nieman Lab in your inbox.

All the latest headlines and insights from the world of digital journalism and publishing. Our daily newsletter is a must-read from newsrooms to executive suites.

Leave this empty

Email address

Name (optional)

Subscribe

We will never sell your address. Prefer a weekly email?

Nearly there — check your email.

We have sent a confirmation link to . Click it and you are on the list. If it has not turned up in a minute or two, it may be in your spam folder.

Subscribe another address

Nieman Lab Anthropic, OpenAI, Google, and Microsoft will make it easier for publishers to opt out of being scraped illustration

Subscribe

Anthropic, OpenAI, Google, and Microsoft will make it easier for publishers to opt out of being scraped

Link datatracker.ietf.org ↗

Posted by Laura Hazard Owen

November 2, 2026

Right now, if you don’t want your site to be scraped by AI companies, you have to block each company’s crawlers individually. On Friday, Anthropic, OpenAI , Google , and Microsoft said they would support a common robots.txt signal that, per OpenAI, “makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.”

Anthropic engineer Kevin Kelley proposed the common signal as part of work for the Internet Engineering Task Force, and Anthropic is already supporting it. OpenAI says it will do so “in the coming months,” and Google and Microsoft say they’ll do so next year.

Meta VP of communications Andy Stone called the common signal “an important step,” writing in a tweet that the company supports “publishers’ ability to express preferences about how their content is used for foundational generative AI training.” Meta is under scrutiny for proposed commercial terms that would allow broad scraping ; last week, it agreed to delay implementation of those terms.

Get Nieman Lab’s Daily Digest

The most important stories about journalism and the future of news, delivered to your inbox every afternoon.

Subscribe

Laura Hazard Owen is the editor of Nieman Lab. You can reach her via email , Bluesky , or Signal (laurahazardowen.38).

November 2, 2026

See more on AI & Journalism

Tags

AI

Andy Stone

Anthropic

Google

Kevin Kelley

meta

Microsoft

OpenAI

robots.txt

The latest from Nieman Lab

Anthropic, OpenAI, Google, and Microsoft will make it easier for publishers to opt out of being scraped illustration

Elon Musk’s Grokipedia is littered with mistakes. But Wikipedia could still learn some things from it.

Bill Adair

The Nieman Lab News-About-News Quiz: October 30, 2026

Joshua Benton

Press Forward gave $20 million to 205 local newsrooms over two years. Here’s how some of them spent it.

Sophie Culpepper

Only 32% of Americans have high confidence that news orgs will report midterm election results accurately, survey finds

Sarah Scire

The New York Times is adding more AI features to its Cooking app, and users aren’t thrilled

Joshua Benton

The latest from Nieman Lab

The Nieman Lab News-About-News Quiz: October 30, 2026

The Nieman Lab News-About-News Quiz: October 30, 2026

Need another reason to read that 47,212th article about paywall strategy? What if it earned you a point ?

Joshua Benton

Only 32% of Americans have high confidence that news orgs will report midterm election results accurately, survey finds

Sarah Scire

Elon Musk’s Grokipedia is littered with mistakes. But Wikipedia could still learn some things from it.

Elon Musk’s Grokipedia is littered with mistakes. But Wikipedia could still learn some things from it.

Grokipedia shows that AI is actually pretty good at finding tidbits in the far corners of the internet.

Bill Adair

Press Forward gave $20 million to 205 local newsrooms over two years. Here’s how some of them spent it.

Press Forward gave $20 million to 205 local newsrooms over two years. Here’s how some of them spent it.

The unrestricted funding helped small outlets plug holes and try new things, but publishers ran up against the limits of what two $50,000 grants could accomplish, especially when it came to hiring.

Sophie Culpepper

The Independent Journalism Atlas looks at “wells” and “springs” in news deserts

Sophie Culpepper

Nieman

Help advance the Nieman Foundation’s mission “to promote and elevate the standards of journalism” by making a donation .

Nieman Foundation for Journalism at Harvard

Walter Lippmann House, One Francis Ave. Cambridge, MA 02138

(617) 495-2237

© 2026 by the President and Fellows of Harvard College

Get Nieman Lab in your inbox.

All the latest headlines and insights from the world of digital journalism and publishing. Our daily newsletter is a must-read from newsrooms to executive suites.

Leave this empty

Email address

Name (optional)

Subscribe

We’ll never sell your address. Prefer a weekly email?

Nearly there — check your email.

We’ve sent a confirmation link to . Click it and you’re on the list.

Close