webscrape.dev

adding web scraping to n8n workflows in 2026

How to call a scraping API from an n8n HTTP Request node, or feed a Crawlee crawler's output back in, and when each approach fits.

Nathan Kessler
Nathan Kessler··Reviewed
3 min read·n8n

Some links on this page are affiliate links. We earn a commission if you sign up – at no additional cost to you. Our editorial assessment is independent and never paid. How we review.

Tools we recommend for n8n

ScraperAPIFreemium

ScraperAPI wraps proxy rotation, headless browsers, and CAPTCHA handling behind a single endpoint. You send a URL and get the rendered HTML back, which keeps it simple for teams that do not want to manage the moving parts themselves.

Web Scraping APIs

ZyteFreemium

Zyte, from the team that maintains Scrapy, offers a scraping API with automatic extraction plus managed proxy and browser handling. It bridges the gap between a raw framework and a fully managed data service.

Web Scraping APIs

CrawleeFree

Crawlee is a crawling and scraping library for Node and Python from Apify. It ships with request queuing, session pooling, and anti-blocking helpers, so more of the hard parts are built in than with a bare framework.

Crawling Frameworks

Notes

n8n is a general workflow-automation tool, not a scraper. It is good at orchestration: triggering on a schedule or webhook, moving data between services, and branching on conditions. For the actual page fetching you point it at a scraping service or a separate crawler process. For most n8n users the cleanest path is an HTTP Request node calling a scraping API such as ScraperAPI or Zyte, because the proxy, browser, and anti-bot handling live outside your workflow. Reach for a standalone Crawlee crawler when a job needs stateful crawling or link-following that a single request cannot express.

n8n handles the orchestration around a scrape: what triggers it, where the data goes, and what happens on failure. It does not do the hard part of fetching a defended page. So the practical question is how to connect n8n to something that does. There are two patterns worth knowing.

Call a scraping API from the HTTP Request node

The most direct route is n8n's HTTP Request node pointed at a scraping API. You send a target URL, and the service returns rendered HTML or structured fields after handling proxy rotation, headless rendering, and CAPTCHA on its side. ScraperAPI fits this shape well: one endpoint, send a URL, get HTML back, with rendering and proxying wrapped behind it. Zyte works the same way and adds automatic extraction, which is a good match if you would rather receive parsed items than raw markup to sift through in n8n.

To wire it up, add an HTTP Request node, set the method and the API's endpoint, put your key in an n8n credential rather than in the node body, and pass the target URL as a parameter. The response flows into the next node, where you map, filter, or write it to a database, sheet, or webhook. This keeps the whole scrape inside a single workflow, which is easy to schedule and easy to debug.

This pattern suits mid-sized jobs where each URL is an independent request. It is the lowest-maintenance option because you are not running any crawler infrastructure yourself.

Run a Crawlee crawler and feed results in

When the work is genuinely a crawl, following links, deduplicating a queue, or holding session state across many pages, a single HTTP call per URL stops being a good fit. Here you run a Crawlee crawler as its own Node or Python process. Crawlee gives you request queuing, session pooling, and anti-blocking helpers, and its Node story is the more mature of the two.

n8n then sits on either end of that process. You can trigger the crawler from an n8n workflow and have it post its dataset back to an n8n webhook when it finishes, or have n8n read the crawler's output from storage on a schedule. Either way, the stateful crawling lives in Crawlee and n8n handles the routing and the downstream steps. Reach for this when the fetch logic is too involved to express as a linear sequence of nodes.

Practical cautions

A few things bite people regardless of which pattern you pick.

Respect the target. Read robots directives and the site's terms before collecting, and do not treat a technical ability to fetch as permission to.

Rate limits apply on both sides. Space out requests so you do not trip the target's defenses or your API plan's ceiling. n8n's iteration and wait steps are the place to add that spacing.

Pagination needs a deliberate loop. Iterate over pages or cursors with a delay between calls, and stop on an empty page rather than a fixed count you guessed at.

Cache and store keys properly. Save fetched pages so reruns do not re-fetch unchanged content, and keep every API key in n8n credentials, never inline in a node. For a wider view of which tool fits which job, see the choosing web scraping tools guide.

Frequently asked

Can n8n scrape a website on its own?
n8n can issue HTTP requests and parse the response, so it can pull simple, static pages. It does not manage proxies, headless browsers, or CAPTCHA handling, so for protected or JavaScript-heavy targets you call a scraping API or run a dedicated crawler and pass the results back in.
Should I use the HTTP Request node or a separate crawler?
Use the HTTP Request node when one URL maps to one call and a scraping API returns the rendered HTML or extracted fields. Use a separate Crawlee crawler when the job needs queuing, link discovery, or session state across many pages, then hand the output to n8n for the downstream steps.
How do I handle pagination in an n8n scraping workflow?
Loop over page URLs or cursors with n8n's iteration nodes, making one API call per page and adding a delay between them. If pagination logic is complex or depends on crawl state, do the crawling in Crawlee and send n8n a finished dataset.
What should I watch out for legally and technically?
Check each target's robots directives and terms of service before you collect anything, keep request rates modest, and cache results so you are not re-fetching the same pages. Store API keys in n8n credentials rather than hardcoding them in nodes.

Weekly briefing – tool launches, pricing shifts, market data.