Open source web scraping tools

Scrapy

Mature Python framework for building and running your own crawlers

Crawlee

Node and Python crawling library with built-in anti-blocking

Automa

Open-source browser extension for no-code workflow automation

Firecrawl

Open-source API that scrapes and crawls sites into LLM-ready data

ScrapeGraphAI

AI-powered API that turns any webpage into structured JSON via prompts

Parsera

No-code AI scraper that turns prompts into structured-data APIs

LLM Scraper

TypeScript library that turns webpages into structured data via LLMs

AgentQL

Natural-language query language for structured web data extraction

Jina Reader

Turns any URL into clean markdown via a simple API prefix

BrowserOS

Open-source Chromium browser with a built-in AI agent

FetchFox

Open-source AI scraper with a hosted API and dashboard

Stagehand

Open source AI browser automation SDK from Browserbase

Browser Use

Open-source browser agent framework plus a paid cloud API

Skyvern

Open source browser automation agent using LLMs and vision

Nanobrowser

Open-source Chrome extension for multi-agent browser automation

Bytebot

Open-source AI agent that automates tasks via a virtual desktop

Agent-E

Open-source LLM agent framework for browser automation

LaVague

Open-source framework for building AI agents to navigate websites

TinyFish

Unified API platform giving AI agents structured access to the web

Cua

Open-source framework for AI agents to control desktop OS sandboxes

Colly

Open-source Go library for building web crawlers and scrapers

Playwright

Open-source browser automation library for Chromium, Firefox, WebKit

Crawl4AI

Open-source Python crawler built for LLM and AI-agent pipelines

Spider

A Rust-based web crawling and scraping API with JS rendering

Katana

Open-source Go web crawler with headless JS rendering and JSON output

Puppeteer

Node.js library from Google for headless Chrome automation

Selenium

Open-source browser automation framework for testing and scraping

Beautiful Soup

Open-source Python library for parsing HTML/XML documents

node-crawler

Node.js crawling library with connection pooling and Cheerio parsing

Nokogiri

Ruby library for parsing and querying XML and HTML documents

Apache Nutch

Java web crawler framework for large-scale, Hadoop-based crawling

WebVoyager

Open-source multimodal web agent with a 643-task benchmark

WebArena

Self-hosted, open-source benchmark for autonomous web agents

Mind2Web

Benchmark of 2,350 tasks across 137 real websites for web agents

BrowserGym

Gym-style API unifying web-agent benchmarks like WebArena

MiniWoB++

Benchmark suite of 100+ simulated web tasks for web agents

AssistantBench

Academic benchmark testing AI agents on realistic web tasks

WebCanvas

Open-source benchmark for evaluating web agents online

GAIA

Benchmark for evaluating general AI agents on reasoning and tool use

Bananalyzer

Open-source Playwright benchmark for evaluating AI web agents

OSWorld

Benchmark for evaluating computer-use AI agents on real tasks

VisualWebArena

Benchmark of 910 visually grounded tasks for multimodal web agents

WebShop

Simulated e-commerce benchmark for training web-shopping agents

Browserless

Cloud headless browsers you drive with Puppeteer or Playwright

Lightpanda

Headless browser engine built in Zig, not a Chromium fork

Steel

Open-source cloud browser API with sessions and stealth for AI agents