Distributed Crawling

Distributed crawling is running one logical crawl across many workers, whether those are processes on one box, containers in a cluster, or serverless jobs. The fetching code barely changes. What changes is that every piece of state a single-process crawler kept in memory now has to be shared, and that is where the work is.

Three coordination problems appear as soon as there is more than one worker. The frontier and the seen-set must be shared, or workers rediscover and refetch the same URLs and the crawl does duplicate work in proportion to worker count. Politeness must be enforced globally rather than per worker, and this is the single most common mistake in the pattern: ten workers each politely limited to one request per second to a host produce ten requests per second at that host, which is not polite and gets the pool blocked. The usual fix is partitioning by host hash, so every URL for a given host lands on the same worker and per-host rate state stays local and correct, at the cost of some imbalance when one host dominates the frontier. Scrapy, Crawlee, Colly, and Apify are all commonly run this way, with the shared state in Redis or a managed queue.

Failure handling forces a second design choice. Distributed queues generally give at-least-once delivery with a visibility timeout: a message reappears if the worker that claimed it dies or runs past the timeout without acknowledging. That is the behavior you want, since the alternative silently drops URLs, but it means the same URL will sometimes be fetched twice. Downstream deduplication on a stable record key therefore stops being an optimization and becomes required for correctness. Set the visibility timeout above your slowest realistic page fetch, or slow pages will be reissued while still in flight.

Autoscaling against queue depth is straightforward to wire up and easy to get wrong, because the constraint upstream is rarely CPU. Purchased proxy concurrency is a global budget the whole fleet shares, so scaling from ten workers to fifty against a contract that allows twenty concurrent sessions converts throughput into connection errors and burned IPs. Enforce politeness and proxy concurrency at the fleet level, expose the current global concurrency as a metric, and size the worker ceiling against the proxy contract you actually bought rather than the throughput you would like to have.