What is node-crawler?
node-crawler (npm package "crawler") is an open-source Node.js web scraping and crawling library maintained by bda-research. It wraps HTTP requests with connection pooling, rate limiting, a priority request queue, automatic retries, charset detection, and request deduplication, and it parses fetched HTML server-side with Cheerio for jQuery-style DOM querying by default. Version 2 is a TypeScript rewrite shipped as native ESM that requires Node.js 22+, and it has no headless browser, so it can't render client-side JavaScript.
Our verdict
It fits Node.js teams that want a lightweight, batteries-included HTTP crawler with built-in retries, pooling, and deduplication rather than assembling that plumbing on top of a bare HTTP client. It has no headless browser, so anything behind client-side JavaScript needs a separate tool, and v2's ESM-only, Node 22+ requirement can rule it out for projects still on CommonJS or older runtimes.
Categories:
Crawling frameworks give engineering teams full control over their pipeline: where it runs, how it scales, and what it collects. The trade is that anti-bot work and infrastructure become your problem rather than a vendor's, which is why they pair well with a proxy network.
