Agent5Seven topics. One clear view.
Agent5

The latest AI news at a glance: seven topics, and a short, plain-English read on the stories. Free, no account.

Product
HomeHow it worksFAQAboutHow we fact-checkBlog
News
All newsModels & ReleasesFunding & DealsBenchmarksAgents & ProductsHardware & ComputePolicy & DramaRobotics
More
AdvertisePressPrivacyTerms
© 2026 Agent5
Agent5
Blog
Agent5
Blog
  1. Home›
  2. News›
  3. Policy & Drama›
  4. Google, Anthropic plan to support AI crawling opt-out standard
Models & Releases20Funding & Deals17Benchmarks13Agents & Products68Hardware & Compute18Policy & Drama36Robotics4
Policy & Drama

Google, Anthropic plan to support AI crawling opt-out standard

SiliconANGLE: Google, Anthropic and others to support new AI crawling opt-out standard

Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block artificial intelligence developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. […] The post Google, Anthropic and others to support new AI crawling opt-out standard appeared first on SiliconANGL

Get smart on it

Major AI developers announced support for a new website crawling standard that would allow website operators to block artificial intelligence developers from using their content to train models. Currently, websites use a text file called robots.txt to manage crawler access, but blocking AI-specific scrapers requires separate entries for each AI provider. The new standard would simplify this process by allowing a single robots.txt entry to block all scrapers collecting training data for generative AI models. The developers plan to incorporate this standard into the existing Robots Exclusion Protocol format and submit it to the Internet Engineering Task Force, with implementation expected to begin in the coming months.

Relevant courses

Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.

Browse AI courses on Maven →
Sources
SiliconANGLE ↗
The biggest AI stories, once a day

One short email a day with the stories that matter. Free. Unsubscribe with one click.

Related news
Policy & DramaOpen story →

As Anthropic finally files for IPO, rising risks aren’t slowing the AI buildout, yet

The other artificial intelligence shoe dropped this week, as Anthropic filed for what surely will be the second blockbuster AI initial public offering of stock this year. Nearly 70,000 words in the filing list a ton of risks, but in short, it’s making unimaginably huge bets on buying computing power, it’s uncertain on how long […] The post As Anthropic finally files for IPO, rising risks aren’t slowing the AI buildout – yet appeared first on SiliconANGLE.

Also reported by TechCrunch

Policy & DramaOpen story →

Anthropic VS code extension on Cursor backdoored for nearly 12 hours

An official extension for Anthropic’s Claude model was backdoored and threatened as many as 5,000 developers' machines, multiple sources reported Friday. Anthropic so far has yet to warn customers of the incident. The compromised package was a VS Code extension available in the marketplace of Cursor, a maker of AI-assisted developer tools, the company said Friday. The malicious version, 2.1.313, subsequently affected “at most 5,000 users,” presumably when they either downloaded it or their syste

Policy & DramaOpen story →

AI can turn security fixes into working exploits before users receive an update

GLM 5.3 turned published-but-unshipped Firefox security fixes into working one-click exploits for six releases, at roughly $1,000 each.

Policy & DramaOpen story →

White House makes Section 232 determination regarding foreign-made robots

A Section 232 investigation by the secretary of commerce found that robotics imports pose a threat to U.S. national security and economic competitiveness. Source: Office of the White House Yesterday, U.S. President Donald J. Trump issued a proclamation stating that the secretary of commerce had submitted a Section 232 report about the national security effects of imports of robotics and industrial machinery, as well as related parts and components. “Based on the facts considered in that in

Policy & DramaOpen story →

A Defense Contractor Gave Their Personal AI Agent Access to a Secure Pentagon System

Sources tell WIRED that an Accenture employee’s personal Claude agent ran on a DOD network for two months before it was identified and the vulnerability could be patched.

Policy & DramaOpen story →

Anthropic is asking Wall Street to believe in AGI

Today on Decoder, I’m joined by Joey Brookhart, an analyst at the chip and AI research firm SemiAnalysis, to help me dig into the story inside the numbers of the Anthropic IPO filing — a document known as an S-1. Anthropic just filed its S-1 to go public on the Nasdaq, and it could start trading as soon as next month at a valuation of anywhere between $1.5 trillion and $2 trillion. That would immediately make it one of the most valuable companies on the planet. But the price