[AI & Journalism
AI](https://www.niemanlab.org/beats/ai-journalism/) [Search & Platforms
Search & Platforms](https://www.niemanlab.org/beats/search-platforms/) [Audience & Creators
Audience & Creators](https://www.niemanlab.org/beats/audience-creators/) [Local News
Local News](https://www.niemanlab.org/beats/local-news/) [Reporting & Production
Reporting](https://www.niemanlab.org/beats/reporting-production/) [Business & Revenue
Revenue](https://www.niemanlab.org/beats/business-revenue/)
About Nieman Lab Subscribe Archives Contact
Nieman Foundation Nieman Fellowships Nieman Reports Nieman Storyboard
Appearance
Auto
Light
Dark
Text size
A
A
A
Get Nieman Lab in your inbox.
All the latest headlines and insights from the world of digital journalism and publishing. Our daily newsletter is a must-read from newsrooms to executive suites.
Leave this empty
Email address
Name (optional)
Subscribe
We will never sell your address. Prefer a weekly email?
Nearly there — check your email.
We have sent a confirmation link to . Click it and you are on the list. If it has not turned up in a minute or two, it may be in your spam folder.
Subscribe another address
Subscribe
Anthropic, OpenAI, Google, and Microsoft will make it easier for publishers to opt out of being scraped
Posted by Laura Hazard Owen
Right now, if you don’t want your site to be scraped by AI companies, you have to block each company’s crawlers individually. On Friday, Anthropic, OpenAI , Google , and Microsoft said they would support a common robots.txt signal that, per OpenAI, “makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.”
Anthropic engineer Kevin Kelley proposed the common signal as part of work for the Internet Engineering Task Force, and Anthropic is already supporting it. OpenAI says it will do so “in the coming months,” and Google and Microsoft say they’ll do so next year.
Meta VP of communications Andy Stone called the common signal “an important step,” writing in a tweet that the company supports “publishers’ ability to express preferences about how their content is used for foundational generative AI training.” Meta is under scrutiny for proposed commercial terms that would allow broad scraping ; last week, it agreed to delay implementation of those terms.
Get Nieman Lab’s Daily Digest
The most important stories about journalism and the future of news, delivered to your inbox every afternoon.
Subscribe
Laura Hazard Owen is the editor of Nieman Lab. You can reach her via email , Bluesky , or Signal (laurahazardowen.38).
November 2, 2026
See more on AI & Journalism
Tags
The latest from Nieman Lab
Bill Adair
The Nieman Lab News-About-News Quiz: October 30, 2026
Joshua Benton
Sophie Culpepper
Sarah Scire
The New York Times is adding more AI features to its Cooking app, and users aren’t thrilled
Joshua Benton
The latest from Nieman Lab
The Nieman Lab News-About-News Quiz: October 30, 2026
Need another reason to read that 47,212th article about paywall strategy? What if it earned you a point ?
Joshua Benton
Only 32% of Americans have high confidence that news orgs will report midterm election results accurately, survey finds
Sarah Scire
Elon Musk’s Grokipedia is littered with mistakes. But Wikipedia could still learn some things from it.
Grokipedia shows that AI is actually pretty good at finding tidbits in the far corners of the internet.
Bill Adair
Press Forward gave $20 million to 205 local newsrooms over two years. Here’s how some of them spent it.
The unrestricted funding helped small outlets plug holes and try new things, but publishers ran up against the limits of what two $50,000 grants could accomplish, especially when it came to hiring.
Sophie Culpepper
The Independent Journalism Atlas looks at “wells” and “springs” in news deserts
Sophie Culpepper
Help advance the Nieman Foundation’s mission “to promote and elevate the standards of journalism” by making a donation .
Nieman Foundation for Journalism at Harvard
Walter Lippmann House, One Francis Ave. Cambridge, MA 02138
(617) 495-2237
© 2026 by the President and Fellows of Harvard College
Get Nieman Lab in your inbox.
All the latest headlines and insights from the world of digital journalism and publishing. Our daily newsletter is a must-read from newsrooms to executive suites.
Leave this empty
Email address
Name (optional)
Subscribe
We’ll never sell your address. Prefer a weekly email?
Nearly there — check your email.
We’ve sent a confirmation link to . Click it and you’re on the list.
Close



