URL Frontier
The URL frontier is the crawler's queue of URLs that have been discovered but not yet fetched. It decides what gets crawled next, and in practice its design shapes crawl behavior more than the parser, the proxy pool, or the browser layer. Two crawlers with identical fetching code and different frontiers produce different datasets.
It carries three responsibilities at once. Prioritization decides the traversal: breadth-first spreads across a site and is what you want for coverage, depth-first drives into one branch and suits a narrow extraction, and score-based ordering ranks by expected freshness or commercial value when the frontier is larger than the budget. Politeness requires per-host sub-queues rather than one global queue, because a single large domain will otherwise fill the queue and monopolize every worker while smaller hosts starve. Deduplication checks each discovered URL against a seen-set before admission, usually after normalization, since session ids and tracking parameters turn one page into thousands of distinct-looking URLs. Scrapy, Crawlee, Nutch, and Katana each ship a version of this, differing mostly in how much of it is pluggable.
The backing store is the thing that changes with scale. An in-process heap is fine for a few hundred thousand URLs on one machine. Past that, or as soon as a second worker exists, the frontier and the seen-set move to Redis or a durable queue so state is shared rather than duplicated. Restart behavior forces the same move: an in-memory frontier means a crashed crawler restarts from seed and re-fetches everything it had already done, which on a multi-day crawl is not recoverable within the schedule. Persisting the frontier and the seen-set turns a crash into a resume.
The practical decision is when to move the frontier out of process, and how to arrange per-host queues so politeness does not cost throughput. The two goals only conflict if you conflate them. One request per second to a single host is slow; one request per second to each of four hundred hosts, drawn round-robin from per-host queues, saturates the concurrency you bought from a proxy vendor while every individual origin sees a light load. If your crawl runs polite and idle at the same time, the frontier is almost always the reason, because breadth of ready hosts, not raw worker count, is what keeps a polite crawl busy.
Tools that handle url frontier
4 tools in the webscrape.dev directory are commonly used for url frontier workflows, spanning crawling frameworks. Each is reviewed independently with pricing and editorial assessment.
Scrapy is a mature, widely-used Python framework for building crawlers. It handles request scheduling, concurrency, and data pipelines, leaving proxies and anti-bot handling to you or an add-on service.
Crawlee is a crawling and scraping library for Node and Python from Apify. It ships with request queuing, session pooling, and anti-blocking helpers, so more of the hard parts are built in than with a bare framework.
Apache Nutch is an open-source, Java-based web crawler framework and a top-level Apache Software Foundation project. It's built for large-scale batch crawling that traditionally runs on Apache Hadoop for distributed fetch, parse, and index jobs, though it can also run on a single machine for smaller crawls. It integrates with Apache Tika for content parsing and can index crawled content into Apache Solr or Elasticsearch, and its plugin-based architecture exposes stable extension points for parsers, HTML filtering, indexing, and scoring. The GitHub repo has roughly 3.3k stars and is Apache-2.0 licensed.
Katana is an open-source web crawling and spidering framework written in Go and built by ProjectDiscovery. It crawls in a standard fast HTTP mode or a headless Chromium mode for JavaScript-rendered pages. It can extract endpoints from JS files, apply regex-based scope rules, and dedupe results by content similarity, with output in plain text, JSON, or JSONL, plus ML-based classification for login, error, and captcha pages. It ships as a CLI binary, a Docker image, and an importable Go library under the MIT license.