
How I built a simple self-healing Scrapy agent
How I built a self-healing Scrapy agent that detects broken selectors after a website change, then has an AI agent repair and test the fix.
Tag archive

How I built a self-healing Scrapy agent that detects broken selectors after a website change, then has an AI agent repair and test the fix.

These four names come up in almost every conversation about scraping with Python, usually presented...

A lot of the work in web scraping happens before you write the scraper, because you need to...

An agent repairs a Scrapy spider after a redesign, then heals a live crawl mid-run through Spidermon and the Scrapy MCP. From our Extract Summit Austin workshop

Scrapy 2.19's RemoteControl extension runs a small HTTP server inside every crawl. What it exposes, how to talk to it, and how Scrapy MCP builds on it.

One Scrapy spider, run with stock Playwright, Patchright and Zyte's remote CDP browser against a bot-challenging test catalogue: what each returned and how to configure them.

TL;DR I built scrapy-jev, a Scrapy pipeline that asks a fast, cheap AI model whether each...

40.6% of popular landing pages need JavaScript to show you anything useful. That is from State of Web...

Two incidents look the same on most scraping dashboards. In the first, one spider starts failing...

When I size up a new website I used to start with headers. Copy them out of DevTools, match the...

A page can contain more JavaScript than HTML and still not need a browser. Often the data is already...

A Scrapy job can exit cleanly after collecting zero products. It can also export 20,000 products with...