Web giant Cloudflare to block AI bots from scraping content by default

undefined 2, 2025

Source: shellypalmer

Cloudflare will now block AI crawlers from accessing websites by default unless owners permit or charge for access, impacting AI developers’ data scraping. Starting Tuesday, new domains can deny AI bots or use a “pay per crawl” model. AI crawlers gather data to train models but reduce traffic and revenue for content creators. OpenAI declined to join Cloudflare’s plan, citing concerns over added middlemen. Experts say this may limit AI training data, affecting chatbot development and model viability long term.

Share on