Web scraping tools with a Python SDK

Scrapy

Mature Python framework for building and running your own crawlers

Evomi

Swiss residential and datacenter proxy network with a scraper API

Byteful (formerly Ping Proxies)

Proxy network provider with four proxy types and a REST API

Mars Proxies

Proxy network with residential, datacenter, ISP, and mobile IPs

Proxy-Seller

Sells residential, mobile, ISP and datacenter proxies

Apify

Cloud platform for running and sharing scraping bots

Databar.ai

No-code GTM data enrichment via 160+ third-party providers

Lobstr.io

No-code cloud platform for prebuilt LinkedIn and Maps scrapers

ScrapeGraphAI

AI-powered API that turns any webpage into structured JSON via prompts

Parsera

No-code AI scraper that turns prompts into structured-data APIs

Kadoa

AI platform for extracting and monitoring structured web data

AgentQL

Natural-language query language for structured web data extraction

JigsawStack

Small AI models via API for web scraping, search, and OCR

Kameleo

Self-hosted anti-detect browser with spoofed device fingerprints

Stagehand

Open source AI browser automation SDK from Browserbase

Skyvern

Open source browser automation agent using LLMs and vision

Agent-E

Open-source LLM agent framework for browser automation

LaVague

Open-source framework for building AI agents to navigate websites

Playwright

Open-source browser automation library for Chromium, Firefox, WebKit

Crawl4AI

Open-source Python crawler built for LLM and AI-agent pipelines

Selenium

Open-source browser automation framework for testing and scraping

Beautiful Soup

Open-source Python library for parsing HTML/XML documents

BrowserGym

Gym-style API unifying web-agent benchmarks like WebArena

MiniWoB++

Benchmark suite of 100+ simulated web tasks for web agents

Bananalyzer

Open-source Playwright benchmark for evaluating AI web agents

WebShop

Simulated e-commerce benchmark for training web-shopping agents

People Data Labs

B2B API for person, company, and job-posting data enrichment