Crawling Frameworks

Crawlee

Node and Python crawling library with built-in anti-blocking

Playwright

Open-source browser automation library for Chromium, Firefox, WebKit

Scrapy

Mature Python framework for building and running your own crawlers

Apache Nutch

Java web crawler framework for large-scale, Hadoop-based crawling

Beautiful Soup

Open-source Python library for parsing HTML/XML documents

Colly

Open-source Go library for building web crawlers and scrapers

Crawl4AI

Open-source Python crawler built for LLM and AI-agent pipelines

Katana

Open-source Go web crawler with headless JS rendering and JSON output

node-crawler

Node.js crawling library with connection pooling and Cheerio parsing

Nokogiri

Ruby library for parsing and querying XML and HTML documents

Puppeteer

Node.js library from Google for headless Chrome automation

Selenium

Open-source browser automation framework for testing and scraping

Spider

A Rust-based web crawling and scraping API with JS rendering