Scraper Middleware

Scraper middleware is the interception layer a crawling framework exposes around every request and response, so behavior can be changed in one place instead of inside each spider. Scrapy splits it in two: downloader middleware sits between the engine and the network, spider middleware sits between the engine and your parsing code. Crawlee exposes pre-navigation hooks alongside a shared ProxyConfiguration and session pool. Colly registers callbacks that fire in a fixed sequence around each fetch, documented as OnRequest, OnError, OnResponseHeaders, OnResponse, OnHTML, OnXML, OnScraped. Different vocabulary, same shape.

The work that belongs there is everything that is true of all requests rather than of one site: proxy selection and rotation, header and TLS fingerprint assembly, retry and backoff policy, response validation, caching, and metrics emission. Soft-block detection belongs here in particular. A challenge page or an empty results shell returned with HTTP 200 is not an error to the transport layer, so unless a middleware inspects the body and marks it failed, the crawl records thousands of successful fetches of nothing. Monitoring products such as ScrapeOps attach at this same slot, which is why the stats they produce are only as good as the validation you put next to them.

Ordering matters more than it looks. Scrapy's documentation, read on 2026-07-27, states that process_request runs in increasing middleware order and process_response runs in decreasing order, and the default DOWNLOADER_MIDDLEWARES_BASE places RetryMiddleware at 550 and HttpProxyMiddleware at 750. A retried request is a copy of the original with its meta dictionary intact, so the proxy chosen the first time rides along with it. Unless something clears that key or the rotator is positioned to reassign it, three retries go out through the same blocked exit and all three fail the same way. This is the classic symptom of a retry component sitting on the wrong side of the rotator: retry counts climb, success does not.

The decision this informs is where the anti-block stack lives. Put block detection, rotation, and backoff in middleware and adding a target becomes a parser plus a config entry. Put them in per-spider code and every new target re-implements them slightly differently, so a change to the backoff curve or a newly recognized challenge signature has to be applied by hand across every crawler you run. Middleware is also testable in isolation, which per-spider logic rarely is.