Cloudflare, 2026년까지 혼합용 AI 크롤러 차단 추진
Cloudflare는 2026년까지 광고 지원 페이지에서 혼합용 크롤러를 차단해, AI 연구소들이 검색용 봇과 학습용 봇을 분리하거나 퍼블리셔에게 콘텐츠 대가를 지불하도록 강제할 예정입니다.
cloudflare announced a september 2026 deadline to block mixed-use crawlers from ad-supported pages unless site owners opt-in, targeting bots that bundle search indexing with ai training. the policy responds to non-human traffic surpassing human traffic and aims to stop ai companies from scraping intellectual property for free. cloudflare is also evolving its pay per crawl marketplace into pay per use, partnering with ceramic.ai and you.com to let publishers charge when content generates value. this forces ai providers to clarify bot intent or establish commercial agreements to access training data.