Free web scraping tools

Scrapy

Mature Python framework for building and running your own crawlers

Crawlee

Node and Python crawling library with built-in anti-blocking

Instant Data Scraper

Free browser extension that auto-detects and exports web page tables

LLM Scraper

TypeScript library that turns webpages into structured data via LLMs

BrowserOS

Open-source Chromium browser with a built-in AI agent

Nanobrowser

Open-source Chrome extension for multi-agent browser automation

Colly

Open-source Go library for building web crawlers and scrapers

Playwright

Open-source browser automation library for Chromium, Firefox, WebKit

Crawl4AI

Open-source Python crawler built for LLM and AI-agent pipelines

Katana

Open-source Go web crawler with headless JS rendering and JSON output

Puppeteer

Node.js library from Google for headless Chrome automation

Selenium

Open-source browser automation framework for testing and scraping

Beautiful Soup

Open-source Python library for parsing HTML/XML documents

node-crawler

Node.js crawling library with connection pooling and Cheerio parsing

Nokogiri

Ruby library for parsing and querying XML and HTML documents

Apache Nutch

Java web crawler framework for large-scale, Hadoop-based crawling

WebVoyager

Open-source multimodal web agent with a 643-task benchmark

WebArena

Self-hosted, open-source benchmark for autonomous web agents

Mind2Web

Benchmark of 2,350 tasks across 137 real websites for web agents

BrowserGym

Gym-style API unifying web-agent benchmarks like WebArena

MiniWoB++

Benchmark suite of 100+ simulated web tasks for web agents

AssistantBench

Academic benchmark testing AI agents on realistic web tasks

WebCanvas

Open-source benchmark for evaluating web agents online

GAIA

Benchmark for evaluating general AI agents on reasoning and tool use

Bananalyzer

Open-source Playwright benchmark for evaluating AI web agents

OSWorld

Benchmark for evaluating computer-use AI agents on real tasks

VisualWebArena

Benchmark of 910 visually grounded tasks for multimodal web agents

WebShop

Simulated e-commerce benchmark for training web-shopping agents