Crawling Ten Million Pages: Frontier Design, Dedup, Politeness and Storage

One request per second to a single host is 86,400 pages a day, so ten million pages from that host takes about 116 days, and adding workers does not move the number. Arithmetic like that, not the fetch loop, is what separates a crawl of ten thousand pages from a crawl of ten million. The loop barely changes between them. What changes is everything around it: which URL the crawler picks next, which ones it refuses to fetch again, how hard it leans on any single server, what it does when a fetch fails in one of the nine interesting ways, and where the bytes end up.

Tutorials stop before that boundary. They show a spider that loads a listing page, follows twenty links and prints titles, which teaches you the library's API and nothing about the failure on day four when the process dies at 6.2 million pages with the queue in memory. Framework documentation has the opposite problem: it describes features accurately and in isolation, so you learn that a scheduler exists without learning what to do with it.

The systems layer is what follows. Frontier design and priority, URL normalisation, dedup with real memory budgets, per-host politeness separated from global throughput, retry policy, checkpointing, and storage layout. Framework choice comes at the end, because it is a consequence of those decisions rather than the starting point. If you are still choosing between a self-hosted crawler, a managed API and a proxy contract, start with how to choose a web scraping tool and come back here once you have decided to run the crawl yourself.

Frontier design decides what actually gets crawled

The frontier is the set of discovered but unfetched URLs, plus the policy that decides which one leaves the queue next. Most of the rest of a crawler is plumbing around that decision.

Two things break if the frontier is a plain FIFO list. The first is ordering. Discovery order is almost never value order, so a breadth-first crawl of a large e-commerce site spends its first several million requests on faceted navigation, pagination, sort permutations and login redirects while the product pages sit behind them. Priority is how you spend a finite request budget on the pages you launched the crawl for. The second is location. A frontier in process memory dies with the process, and at ten million pages the process will die.

At this scale the frontier belongs in Redis, a database, or a partitioned log, and it should hold fingerprints rather than full URLs. Ten million fetched pages routinely discover an order of magnitude more URLs than they fetch, so the frontier is usually the larger of the two data structures.

The structural answer that has survived since the Mercator crawler design is a two-level queue: front queues that hold priority classes, and back queues with one queue per host. A worker pulls from a front queue by priority, but the URL it gets is routed through the back queue for its host, which is where the politeness delay is enforced. That split is what lets you run 500 concurrent requests globally while never sending two simultaneous requests to the same server.

Frameworks expose this differently. Scrapy implements the frontier as a priority queue inside its scheduler, with memory and disk queue classes and a SCHEDULER_PRIORITY_QUEUE setting. In the settings reference as it stands on 27 July 2026, that setting defaults to DownloaderAwarePriorityQueue, the queue that spreads pending requests across domains rather than draining one at a time, with ScrapyPriorityQueue as the other available type. Older write-ups that tell you to switch the downloader-aware queue on by hand are describing an earlier default. Crawl4AI exposes the same concept as deep-crawl strategies, with BFSDeepCrawlStrategy, DFSDeepCrawlStrategy and a BestFirstCrawlingStrategy that orders the frontier by a scorer you supply, along with a filter chain and a max_pages cap. node-crawler offers explicit priorityLevels with a per-request priority, which is the same idea with a smaller surface.

Normalise URLs before you deduplicate them

Deduplication is only as good as the string you hash. Before anything reaches the seen-set, put every URL through the same normaliser:

• Lowercase the scheme and host, punycode the host, drop the default port.
• Resolve relative references and dot segments to an absolute path.
• Drop the fragment. It never reaches the server.
• Normalise percent-encoding to a canonical case and unescape unreserved characters.
• Decide a trailing-slash rule and apply it everywhere.
• Strip tracking parameters (utm_*, gclid, fbclid) and session identifiers, then sort the remaining query parameters.

That last rule is where most of the duplication lives. Parameter order and tracking junk turn one page into dozens of distinct strings, and a crawler that treats them as distinct will happily fetch the same document forty times.

Keep crawl-time canonicalisation separate from record-time canonicalisation. rel=canonical in the page head is a publisher hint about which document should win in an index. A crawler should record it and use it when choosing which stored document represents the page, but obeying it during the crawl means trusting an arbitrary site to reshape your frontier. Record it, resolve it later.

Then cap the traps. Infinite calendars, faceted navigation with combinatorial parameter sets, session identifiers in the path, and mirrored subdomains can each generate unbounded URL space that passes every normalisation rule you wrote. The defences are blunt and effective: a maximum path depth, a maximum number of query parameters, a per-host page budget, and a scope regex. Katana makes scope a first-class flag: -fs defaults to rdn (root domain name), -cs and -cos take in-scope and out-of-scope regexes, and -d caps depth at 3 unless you raise it. Crawl4AI's documentation makes the same point from the other direction, warning that a max_depth above 3 grows the crawl exponentially.

Deduplication: URLs first, content second

Two different questions get conflated under "dedup", and they need different data structures.

Have I fetched this URL? This is a membership test over the normalised URL, and it runs on every discovered link, so it must be cheap. Hash to a fixed-width fingerprint and store the fingerprint. An exact set of 16-byte fingerprints is 160 MB of raw keys per 10 million URLs before any container overhead, and Python or JavaScript hash-set overhead can multiply that by two or three. A Bloom filter is the standard way to keep the number flat: the usual sizing is roughly 10 bits per element for a 1% false-positive rate, which is about 12 MB per 10 million URLs and about 1.2 GB per billion. That is arithmetic on the standard formula, not a measurement of any particular implementation.

The catch is what a false positive means here. The filter reports "seen" for a URL you never fetched, and the crawler silently skips a real page with no error anywhere. So use the Bloom filter as a pre-filter in front of an exact store rather than as the store itself: negatives are certain, positives get checked against Redis or a key-value table. Below a few hundred thousand URLs, skip the filter. An ordinary in-memory set is simpler and fits.

Scrapy's dupe filter works on request fingerprints and, with a job directory configured, persists them to disk so the seen-set survives a restart. Crawlee's RequestQueue deduplicates on a unique key derived from the request. Colly tracks visited URLs in its storage layer, which can be swapped for the Redis backend the project documents for distributed crawls. node-crawler has a skipDuplicates option, which is the right feature for a single-process crawl and the wrong one to lean on across a fleet.

Have I seen this content? URL dedup does nothing about the same document served at five addresses, or a soft 404 that returns 200 with a generic body. Hash the normalised response body (SHA-256 over the decoded bytes, with volatile elements like CSRF tokens and timestamps stripped if you can identify them) and treat the hash as the document identity. Byte-identical duplicates fall out immediately, and the hash doubles as your object-storage key.

Near-duplicates need more. Two product pages that differ only in a rendered timestamp have different SHA-256 values and identical value to you. SimHash or MinHash over shingles of the extracted text gives a similarity measure you can threshold. This is worth building when your crawl covers many hosts publishing syndicated content, and not worth building for a scoped crawl of one retailer.

Politeness is per host; throughput is global

These are separate budgets, and conflating them is the most common way a large crawl gets a team blocked.

Global throughput is a function of your infrastructure: workers, connections, bandwidth, and how many IPs you have. Per-host politeness is a function of the target's capacity, and it does not improve when you add workers. One request per second to a single host is 86,400 pages a day. Ten million pages from one host at that rate is roughly 116 days. That number, not your worker count, decides whether the crawl is possible.

The shape of the target list matters more than the total. Ten million pages spread across 50,000 hosts is 200 pages per host. At one request per second per host with 200 hosts in flight, that is 200 requests per second in aggregate and roughly 14 hours of wall clock. Same total, entirely different system. Work out the per-host distribution before you size anything else.

For the rate itself, start at one request per second and let the server's own signals move it. Rising response latency, 429s and 503s are the target telling you to back off, and a crawler that responds to them stays welcome far longer than one that runs a fixed rate. Check robots.txt for a crawl-delay directive, but understand what it is: not part of the original standard, and interpreted differently by everyone. Per Wikipedia's robots.txt entry, Google ignores it, Bing has defined it as a window of 1 to 30 seconds during which BingBot will access the site once, and Yandex reads it as the number of seconds between visits.

The framework knobs map onto this split directly. Scrapy separates CONCURRENT_REQUESTS, default 16, from CONCURRENT_REQUESTS_PER_DOMAIN, which the settings reference on 27 July 2026 gives as 1 (fallback: 8), and adds DOWNLOAD_DELAY, given as 1 (fallback: 0), with AUTOTHROTTLE over the top. Read the current defaults rather than a tutorial's, because they have moved. Scrapy's broad-crawl page states the defaults are optimised for crawling specific sites, and suggests raising global concurrency toward 100, then tuning until CPU sits around 80 to 90 per cent. Colly's LimitRule takes a DomainGlob with Parallelism, Delay and RandomDelay, so politeness is expressed per domain pattern. node-crawler splits maxConnections from rateLimit and supports multiple named limiters, which is how you give each proxy or host its own budget.

None of this is a substitute for the network layer. Politeness keeps you from harming a server; it does not get you past a rotating proxy requirement or an anti-bot system that fingerprints your TLS stack. Those are a separate problem, covered in why scrapers get blocked and solved with a proxy network or browser infrastructure underneath the crawler rather than with a slower crawl.

Retries: transient, terminal, and the third category

A retry policy that treats all failures alike either burns the budget on dead URLs or drops pages that would have worked on the second attempt. Sort failures into three buckets.

Transient. Connection resets, read timeouts, DNS failures, 429, 502, 503, 504. Retry with exponential backoff and jitter, cap at three or four attempts, and apply the backoff at the host level rather than the URL level. If a host returns 503 to one request it will return 503 to the next fifty, so a per-host circuit breaker that pauses the whole back queue does more good than fifty independent retry timers. node-crawler exposes retries and retryInterval per request; Scrapy ships a retry middleware with a configurable status list.

Terminal. 404, 410, robots-disallowed, unparseable content type, response over your size cap. Record the outcome and never enqueue the URL again. The record matters: a URL that silently disappears from the crawl is indistinguishable from a bug.

Blocked. A 403 with a challenge page, a CAPTCHA interstitial, or a 200 carrying a "verify you are human" body is not a transient failure and retrying it at the same rate from the same IP achieves nothing except confirming the block. Route these to a different path: a new session, a different proxy type, or a managed web scraping API for the subset of hosts that need it. If a single vendor is your only route to those hosts, multi-vendor failover is the shape of the fix.

Everything that exhausts its retries goes to a dead-letter store with the last status, the error and the attempt count. That table is the crawl's diagnostic surface. A crawl without one is a crawl where you cannot answer why coverage came in at 94%.

Checkpointing, so a restart is not a re-crawl

Mark a URL done only after its document is durably stored, never when the fetch returns. That rule gets broken constantly.

Track each URL through explicit states, at minimum enqueued, in-flight, stored and done, with a lease timestamp on in-flight so a worker that dies releases its claim after a timeout instead of stranding the URL. If you mark URLs done at fetch time, a crash between fetch and write loses documents that the dupe filter now believes are complete, and no restart will ever fetch them again. That failure is quiet, which is what makes it expensive.

Checkpoint cadence is a straightforward trade: more frequent checkpoints cost write throughput, less frequent ones cost replay time. Batching state updates every few hundred completed URLs is usually the right order of magnitude, with the raw documents written before the state batch commits.

The frameworks give you different amounts of this for free:

• Scrapy's job directory persists the scheduler's pending requests, the dupe filter's visited requests, and spider key-value state between batches. Its documentation is explicit that this is the supported pause-and-resume mechanism.
• Crawlee's storage clients decide the answer, and the Python docs as of 27 July 2026 list five. FileSystemStorageClient writes queues, datasets and key-value stores to disk; SqlStorageClient (SQLite, PostgreSQL, MySQL, MariaDB) and RedisStorageClient persist to a database, which is also how you share one queue across machines; ApifyStorageClient uses Apify's platform. MemoryStorageClient, in the documentation's words, loses all data when the program terminates. A custom storage client interface is documented if none of those fits.
• Colly documents a Redis storage backend that holds both the request queue and the visited set, which makes a restart or a second machine a configuration change rather than a rewrite.
• Apache Nutch never has the problem in the same form, because its unit of work is a batch cycle over a CrawlDb rather than a long-lived process. State is on disk between generate, fetch, parse and update steps by construction.

Storage: raw documents and parsed records are two systems

Keep the bytes. Parsers change constantly, sites change their markup, and the extraction rule you wrote in March will be wrong in July. If you kept only parsed records, fixing it means re-crawling ten million pages. If you kept the raw documents, it means re-running a parser over object storage, which is hours instead of months and costs no requests against anyone's server.

A workable layout:

• Raw layer. Object storage, one compressed object per document or per batch, keyed by content hash. Content-hash keys make writes idempotent and give you byte-identical dedup for free. Partition by fetch date and host so a re-parse can be scoped. WARC, standardised as ISO 28500, is the archival container if you want request and response headers preserved alongside the body; Common Crawl documents it as the format for its own multi-billion-page archives, with WAT and WET files as derived metadata and text extractions. For most commercial crawls, gzipped JSONL or one Parquet file per batch is simpler and enough.
• Parsed layer. Columnar files (Parquet) or a warehouse table, partitioned the same way, with every row carrying a pointer to its raw object, the fetch timestamp and the parser version. The parser version field is what makes selective re-parsing possible: when you fix a selector, you re-run only the rows produced by the broken version.
• State layer. The frontier, the seen-set and the dead-letter table. Small, hot, transactional, and separate from the other two.

Size it before you build it. At an assumed 100 KB per raw HTML document, ten million pages is about 1 TB uncompressed and roughly 200 to 250 GB gzipped at typical HTML compression ratios. Rendered pages from a headless browser run larger, sometimes several times larger, so if the crawl needs JavaScript execution, measure your own corpus rather than trusting that figure. The storage bill is small at this scale. The cost that hurts is reading the whole raw layer back for a full re-parse, which is what the partitioning is there to reduce.

One thing to avoid: storing ten million raw documents as rows in the same transactional database that holds your frontier. It works at a hundred thousand and degrades badly past a few million, usually right when you have the least appetite for a migration.

Choosing the framework last

Once the decisions above are made, framework selection is mostly a matter of which one already implements your answers. All of these are crawling frameworks in the directory, and the table compares them on the architectural axes rather than on features.

Where each one goes wrong:

Scrapy asks for Python in production, and it asks for pages that do not need a real browser, since the Playwright integration is a bolted-on cost rather than the native path. Anti-bot handling is not in the box either; it does none. For everything else at this scale in Python it remains the reference, and the Scrapy vs Crawlee comparison covers the split by stack.

Crawlee has the awkward property that its Node implementation is the older and more complete one, so a Python-first team picking it is picking the younger runtime. Where it earns its place is a crawler that already lives inside a Node service and wants request queuing, session pooling and anti-blocking helpers without assembling them.

Colly fetches over HTTP with no headless browser in the box, so a client-rendered site returns an empty shell and JavaScript-heavy targets are out. Server-rendered targets where throughput per machine matters are its home ground. The README's feature list claims "Fast (>1k request/sec on a single core)", which is the project's own number, unqualified as to page size or target, and not one we have measured.

Apache Nutch assumes you already run a JVM cluster. The batch cycle and the Hadoop data structures underneath it are heavy for a crawl that fits on one machine. The 1.x line is the live one: the project's download page on 27 July 2026 lists 1.22, released 20 July 2025, as the current release and points to the Apache archives for anything older. That cadence is slow by framework standards, which is reassuring in a component you want stable for years and painful when you need a fix quickly. Nutch fits search-index-shaped crawls of hundreds of millions of pages where the cluster already exists.

Katana comes from ProjectDiscovery and is built for scoped endpoint discovery on a target, with regex scope control, JavaScript endpoint parsing and JSONL output. Excellent for mapping a site, poor as the engine behind a long-running production pipeline that needs a persistent frontier.

node-crawler keeps everything in one process: its priorityLevels (default 10, with a per-request priority defaulting to 5), its named rate limiters, its skipDuplicates flag and its retry counters. A second machine shares none of that state, so distribution is off the table. Two other constraints are worth knowing before you commit: the v2 README states that Crawler v2 is native ESM with no CommonJS export, and that it requires Node.js 22 or above. Hundreds of thousands of pages in a self-contained Node crawler, yes. Tens of millions, no.

Crawl4AI orients its output toward clean Markdown for LLM and agent pipelines, so the raw document tree is not what you get back. Its deep-crawl strategies and scorers are a genuine frontier abstraction, and the docs name BestFirstCrawlingStrategy as the recommended one, but the persistence layer is yours to build. Check its robots.txt setting explicitly before any crawl you would be willing to describe in public; the defaults on extraction-first tools are not always the conservative ones.

Spider hands the frontier to a hosted API, which removes exactly the control this guide is about. That is a bad trade when the crawl policy is the product and a good one when the crawl is a means to an end and engineering time costs more than the invoice. Pricing on hosted crawl APIs moves often enough that any number here would be stale; check the vendor's page and read what a scraping API credit actually costs before committing to volume.

What to build first

In order, because the order is what keeps the rewrites small:

1. The URL normaliser and the seen-set. Everything downstream depends on the identity of a page, and changing that definition later invalidates the crawl.
2. The external frontier. Redis or a table, with priority and per-host routing. This is the piece that makes the crawl restartable.
3. The politeness layer. Per-host limits distinct from global concurrency, with backoff wired to the server's signals.
4. The raw storage path. Write the bytes, keyed by content hash, before you write a single parser.
5. State transitions and the dead-letter table. Done means stored, and everything that failed is visible.
6. Parsers. Last, versioned, and re-runnable over the raw layer.

The frameworks in the table above all give you some of steps 2 through 4. None of them gives you step 1, and none of them decides step 5 for you. That is the actual work in a large crawl, and it is why the choice between Scrapy and Colly matters much less than the choice of where the frontier lives. For a wider view of how the crawler fits with proxies, unblockers and extraction, see the web scraping stack in 2026.