B
Oct 30, 2026Blocking AI crawlers: robots.txt vs Cloudflare vs Anubis, and what each one actually does
Three tools, three different jobs. From fifteen days of our own logs: which crawlers ever fetch robots.txt, which ones you can identify at all, and what a proof
Oct 30, 20268 min read0 reactions0 comments