Is the web for humans or AI bots? Cloudflare sets a new default
cloudflare launched granular ai controls to block training/agent bots on ad-supported pages by default, forcing bot operators to disclose their purpose.
cloudflare introduced a new bot management framework allowing site owners to toggle access for search indexing, model training, and user agents independently, rather than applying a blanket block. the system defaults to denying training and agent access on ad-supported pages starting september 15, 2026, reserving that traffic for human visitors, while still permitting search crawlers. cloudflare ceo matthew prince cited that bot traffic recently surpassed human traffic for the first time as a driver for these stricter controls. this move pressures ai labs to stop bundling training functionality with search crawlers, forcing greater transparency about how content is used. enterprise users gain botbase, a dashboard database detailing known bots and their specific content usage to aid in enforcement.