Cloudflare AI Crawl Control: Monitoring and Managing AI Crawlers
I run this blog, so I want to know which automated services request its pages and whether they follow my access rules. Some crawlers index pages for search. Others collect content for artificial intelligence (AI) training or retrieval, and a chatbot may answer a question without sending the reader to the source.
Cloudflare AI Crawl Control, formerly called AI Audit, reports AI crawler activity and lets site owners apply crawler-specific allow or block policies. This post explains the difference between monitoring, voluntary robots.txt rules, and enforcement at Cloudflare's edge.