Web crawling tools

Zyte

Scraping API and cloud platform from the team behind Scrapy

Scrapy

Mature Python framework for building and running your own crawlers

Crawlee

Node and Python crawling library with built-in anti-blocking

Geonode

Proxy network and scraping API with flat per-request pricing

Octoparse

No-code scraper that turns web pages into structured data

Apify

Cloud platform for running and sharing scraping bots

ParseHub

Desktop point-and-click scraper for JS-heavy sites, exports JSON/CSV

Simplescraper

Chrome extension and cloud service for no-code scraping recipes

Stabler

Node-graph builder for JS-rendered scrapers with anti-bot bypass

No-Code Scraper

No-code AI scraper that exports data to CSV, Excel, or JSON

Dexi.io

No-code platform for e-commerce price and assortment monitoring

Web Scraper

Free Chrome extension for visual scraping, paid cloud platform

ScrapeStorm

Desktop visual scraper with AI field detection and flowchart mode

Data Miner

Chrome/Edge extension for point-and-click scraping to CSV/Excel

Scrapeless

Cloud scraping API and proxy network billed only on success

Olostep

Web data API with built-in JS rendering, proxies, and CAPTCHA solving

Crawlbase

Web scraping API platform with JS rendering and residential proxies

Firecrawl

Open-source API that scrapes and crawls sites into LLM-ready data

ScrapeGraphAI

AI-powered API that turns any webpage into structured JSON via prompts

Diffbot

Computer vision and ML API that extracts structured web data

FetchFox

Open-source AI scraper with a hosted API and dashboard

Tavily

Real-time web search, extract, and crawl API for AI agents

Linken Sphere

Desktop antidetect browser for running isolated multi-account profiles

Gumloop

No-code AI agent builder with web scraping and crawling nodes

Colly

Open-source Go library for building web crawlers and scrapers

Crawl4AI

Open-source Python crawler built for LLM and AI-agent pipelines

Spider

A Rust-based web crawling and scraping API with JS rendering

Katana

Open-source Go web crawler with headless JS rendering and JSON output

Puppeteer

Node.js library from Google for headless Chrome automation

Selenium

Open-source browser automation framework for testing and scraping

node-crawler

Node.js crawling library with connection pooling and Cheerio parsing

Nokogiri

Ruby library for parsing and querying XML and HTML documents

Apache Nutch

Java web crawler framework for large-scale, Hadoop-based crawling

DataForSEO

Pay-as-you-go REST APIs for SERP, keyword, and backlink data

Lightpanda

Headless browser engine built in Zig, not a Chromium fork