#aicrawler

Ten of 33 robots.txt cells let a blocked URL through and Codex clipped the back of a 34 KiB AGENTS.md. All 219 runs exited 0. Which parser gave your verdict?

GPTBot and ClaudeBot don't render JS. Here's a curl experiment proving client-side pages are invisible to AI search, plus how to fix it server-side.