Deduplication

Deduplication is the machinery that stops a crawl from fetching the same thing twice and from storing the same record twice. Those are two different problems at two different layers, and a system that solves only one of them still ships duplicates.

URL-level dedup starts with normalization, because hashing raw URLs catches almost nothing. Strip tracking parameters such as utm_ and fbclid, sort the remaining query keys into a stable order, resolve relative and dot segments, lowercase the host while leaving the path case alone, and make one explicit decision about trailing slashes and default ports. Only then hash. Scrapy's default RFPDupeFilter works this way, filtering "based on the REQUEST_FINGERPRINTER_CLASS setting", which is the seam you override when a target puts meaningful state in a header or a POST body. The seen-set behind it is an in-memory structure at small scale and moves to a shared Redis set once several workers share a frontier, or to a Bloom filter once the entry count passes tens of millions.

Content-level dedup handles the same record arriving under different URLs, which normalization cannot reach: a product under two category paths, a listing syndicated across mirrors, a session id baked into a path segment. This needs a hash over the extracted record rather than over the page, computed on the fields you actually care about. Exact hashing then stops working the moment boilerplate is included, because a rotating ad slot or a "last updated 3 minutes ago" string changes the digest on every fetch. Near-duplicate detection is the answer: simhash or minhash over shingles gives a similarity score instead of a binary match, and you set a threshold.

Probabilistic structures trade memory for a controlled error, and the direction of that error matters. A Bloom filter answers "definitely not seen" or "probably seen", so its false positives are dropped URLs rather than duplicate ones. That is a silent coverage bug: pages never fetched, no error logged, a crawl that looks complete. At a 1 percent false-positive rate over 100 million URLs the expected loss is around a million pages. So the decision splits on volume and stakes. Under a few million URLs an exact Redis set costs little and removes the whole class of problem. Above that, size a Bloom filter to a rate you can defend, keep any high-value segment in an exact set anyway, and set the near-duplicate threshold empirically against a labeled sample, since a threshold tight enough to collapse genuine repeats will merge distinct records if you push it one notch further.