Cloudflare is blocking mixed-use AI crawlers by 2026
cloudflare will block mixed-use crawlers from ad-supported pages by 2026, forcing ai labs to separate training bots from search or pay publishers for content.
cloudflare announced a september 2026 deadline to block mixed-use crawlers from ad-supported pages unless site owners opt-in, targeting bots that bundle search indexing with ai training. the policy responds to non-human traffic surpassing human traffic and aims to stop ai companies from scraping intellectual property for free. cloudflare is also evolving its pay per crawl marketplace into pay per use, partnering with ceramic.ai and you.com to let publishers charge when content generates value. this forces ai providers to clarify bot intent or establish commercial agreements to access training data.