Incremental Crawling

Incremental crawling means re-fetching only the pages likely to have changed since the last run, instead of re-crawling the full set on every cycle. On a recurring crawl of any size this is the difference between a bandwidth bill that scales with catalog size and one that scales with change rate, which is usually a small fraction of it.

Change signals are worth ranking by cost. Conditional requests are the cheapest: send If-Modified-Since with the timestamp you stored, or If-None-Match with the stored ETag, and a well-behaved origin answers 304 Not Modified with no body. You still pay for the round trip and, if you route through a metered proxy, whatever that provider bills for the request, but you skip the payload and the parse. Next cheapest is a sitemap with lastmod values, which lets one fetch triage thousands of URLs, though the values are self-reported and some publishers stamp them on every build. Last is content hashing, and the hash must cover the extracted record rather than the raw HTML. Raw-page hashes change on every request because of rotating ad slots, CSRF tokens, "viewed 12 minutes ago" counters, and build ids, so they report change constantly and save nothing.

Layered on top of the signals is adaptive refresh scheduling: hold a per-URL interval, shorten it when a page turns out to have changed, lengthen it when it has not, with a floor and a ceiling. A product page whose price moves daily earns a daily check, an archived listing that has been stable for six months drops to monthly, and the total request budget buys much more freshness than a uniform sweep would. Hosted crawlers such as Spider Cloud, Crawlee, Browse AI, and Apify expose some mix of scheduling and change tracking, but the change definition still has to be yours, because only you know which fields in the record actually matter.

The trap to plan for is the origin that ignores conditional headers and returns 200 with a byte-identical body every time. This is common enough that request-level signals cannot be trusted alone. Detect it per host by comparing the extracted-record hash against the response status, and where the two disagree, fall back to hashing and stop trusting the ETag for that host. The decision this supports is a concrete one: for each host, which signal you rely on, and what per-URL refresh interval you derive from its measured change rate rather than from a guess.