How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python)
Before you decide what to do about AI crawlers, measure them. A Python script that classifies every request in your access logs as search crawler, AI training crawler, AI search, user-triggered AI fetch or other, then reports requests, bandwidth and top paths for each.



