Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare has announced new default settings that will block web crawlers used simultaneously for search, AI agents, and training from accessing pages with ads, starting September 15, 2026. The policy aims to address a situation where website owners want their content discoverable through search and AI services but seek protection against having their intellectual property used for free. The change matters because it affects how AI companies can access web content for training and powering their services, while giving publishers new tools to charge AI companies for using their content through mechanisms like Pay Per Crawl and Pay Per Use. This reflects broader concerns about balancing AI model development with publisher compensation as bot traffic has recently surpassed human traffic on the internet.