AI Crawlers List: Training vs. Search User Agents
Identify AI crawler user agents and choose robots.txt rules that block training crawlers without unintentionally blocking search crawlers.
Tag archive
Identify AI crawler user agents and choose robots.txt rules that block training crawlers without unintentionally blocking search crawlers.
Three tools, three different jobs. From fifteen days of our own logs: which crawlers ever fetch robots.txt, which ones you can identify at all, and what a proof

We gave 45 crawler situations to Sonnet and Haiku. Bare bot names fooled both, and the smaller model also slipped on robots.txt. Each trap and its fi…
Originally published at forgemesh.io. Open your server logs and count the humans. On most sites it...
Originally published at forgemesh.io. Last night Pay-per-Crawl was a WordPress plugin that started...
Originally published at forgemesh.io. Someone on TikTok asked us for a WordPress plugin. We had the...

No rule changes this update, but there is a lot happening in the world of AI crawlers and bots that every site owner should know about.
We pulled 24 hours of real Cloudflare AI Crawl Control data for a mid-sized content site. Applebot alone outweighs Googlebot 12 to 1, and the crawler labeled 'AI' spends real bandwidth on files that aren't content at all.

Originally published at norvik.tech Introduction Explore the mechanics and implications...
Cloudflare Bot Preference Sync will write your robots.txt from 21 August 2026 - and it...

AI agent requests are rising rapidly, while referral traffic remains small, forcing marketers to separate crawling from human visits.

Sorting the 15 AI crawlers by what blocking each actually costs you — not another copy-paste list. Corrects the two token mistakes nearly everyone makes about Google-Extended and Applebot-Extended, and splits bots into the four categories that map to real visibility and training tradeoffs.