Self-hosted web scraping tools

Scrapy

Mature Python framework for building and running your own crawlers

Crawlee

Node and Python crawling library with built-in anti-blocking

Proxidize

Managed mobile and residential proxies with a cloud dashboard and API

Rayobyte

Proxy network with residential, ISP, and datacenter IPs

Automa

Open-source browser extension for no-code workflow automation

Axiom.ai

No-code Chrome extension for browser automation and scraping

WebHarvy

Windows desktop app for visual, no-code web scraping

Firecrawl

Open-source API that scrapes and crawls sites into LLM-ready data

ScrapeGraphAI

AI-powered API that turns any webpage into structured JSON via prompts

Parsera

No-code AI scraper that turns prompts into structured-data APIs

LLM Scraper

TypeScript library that turns webpages into structured data via LLMs

Kadoa

AI platform for extracting and monitoring structured web data

Jina Reader

Turns any URL into clean markdown via a simple API prefix

BrowserOS

Open-source Chromium browser with a built-in AI agent

Linkup

Search, fetch, and research API for AI applications

FetchFox

Open-source AI scraper with a hosted API and dashboard

Kameleo

Self-hosted anti-detect browser with spoofed device fingerprints

Nstbrowser

Cloud antidetect browser bundling proxies and CAPTCHA solving

Stagehand

Open source AI browser automation SDK from Browserbase

Browser Use

Open-source browser agent framework plus a paid cloud API

Skyvern

Open source browser automation agent using LLMs and vision

Nanobrowser

Open-source Chrome extension for multi-agent browser automation

Bytebot

Open-source AI agent that automates tasks via a virtual desktop

Agent-E

Open-source LLM agent framework for browser automation

Notte

Browser automation platform with AI agents and structured scraping

LaVague

Open-source framework for building AI agents to navigate websites

Cua

Open-source framework for AI agents to control desktop OS sandboxes

Colly

Open-source Go library for building web crawlers and scrapers

Playwright

Open-source browser automation library for Chromium, Firefox, WebKit

Crawl4AI

Open-source Python crawler built for LLM and AI-agent pipelines

Spider

A Rust-based web crawling and scraping API with JS rendering

Katana

Open-source Go web crawler with headless JS rendering and JSON output

Puppeteer

Node.js library from Google for headless Chrome automation

Selenium

Open-source browser automation framework for testing and scraping

Beautiful Soup

Open-source Python library for parsing HTML/XML documents

node-crawler

Node.js crawling library with connection pooling and Cheerio parsing

Nokogiri

Ruby library for parsing and querying XML and HTML documents

Apache Nutch

Java web crawler framework for large-scale, Hadoop-based crawling

WebVoyager

Open-source multimodal web agent with a 643-task benchmark

WebArena

Self-hosted, open-source benchmark for autonomous web agents

Mind2Web

Benchmark of 2,350 tasks across 137 real websites for web agents

BrowserGym

Gym-style API unifying web-agent benchmarks like WebArena

MiniWoB++

Benchmark suite of 100+ simulated web tasks for web agents

AssistantBench

Academic benchmark testing AI agents on realistic web tasks

WebCanvas

Open-source benchmark for evaluating web agents online

GAIA

Benchmark for evaluating general AI agents on reasoning and tool use

Bananalyzer

Open-source Playwright benchmark for evaluating AI web agents

OSWorld

Benchmark for evaluating computer-use AI agents on real tasks

VisualWebArena

Benchmark of 910 visually grounded tasks for multimodal web agents

WebShop

Simulated e-commerce benchmark for training web-shopping agents

Browserless

Cloud headless browsers you drive with Puppeteer or Playwright

Lightpanda

Headless browser engine built in Zig, not a Chromium fork

Steel

Open-source cloud browser API with sessions and stealth for AI agents

Anchor Browser

Cloud browser infrastructure for AI agents to act on websites