Open-source crawlers did not lose. The blocking layer moved out from under them.

Scrapy shipped 2.17.0 on 7 July 2026. The release notes give three highlights: "Security bug fixes", "HTTP/2 and SOCKS proxy support for HttpxDownloadHandler", and "Improved settings for changing allowed TLS versions". Underneath those sit a new HTTPX_HTTP2_ENABLED setting and a pair of settings, DOWNLOAD_TLS_MIN_VERSION and DOWNLOAD_TLS_MAX_VERSION, replacing the deprecated DOWNLOADER_CLIENT_TLS_METHOD.

Read the list again. Not one item is about crawling. Nothing about scheduling, nothing about deduplication, nothing about parsing or item pipelines. Every headline change concerns the transport: which protocol version you speak, which TLS versions you will negotiate, which proxy scheme you can route through. A framework that has been maintained since 2008 spent its newest release working on the connection, because that is where the work is.

The framing that misses the mechanism

The recurring story is that managed scraping APIs are replacing open-source crawlers. Buy the endpoint, skip the framework. It is tidy, and it sends teams to the wrong place with their budget.

Open-source crawlers did not lose a competition. Nothing they do got worse. What happened is that a layer they never owned, the act of reaching the origin at all, turned into a specialised and permanently maintained job, and that job now dominates the cost of the operation. The framework still does its job well. That job is no longer the expensive one.

What was solved, and stayed solved

Request scheduling, politeness and rate limiting, frontier management, deduplication, retry semantics, concurrency, item pipelines, HTML parsing: all of it was worked out years ago, all of it is free, and none of it has regressed.

Scrapy has been the reference implementation of that set in Python for most of two decades and still ships on a steady cadence: 2.15.0 in April 2026, 2.16.0 on 19 May, 2.17.0 on 7 July, according to its own GitHub release history. Crawlee covers the same ground for Node, with v3.17.0 tagged on 4 June 2026. Beautiful Soup has been turning malformed HTML into a queryable tree since before the anti-bot industry existed, and does not need to change.

Colly is the interesting case. Its most recent tagged release is v2.2.0 from 27 March 2025, which looks like abandonment until you read the commit log: it has merges from June 2026, including a fix for HTTP status codes at 203 and above being wrongly treated as errors. A Go library with a settled API does not need a tag every quarter. It needs someone merging correctness fixes, and someone is.

None of that looks like a category in decline. It looks like a solved problem behaving normally, releasing slowly because the requirements stopped moving. The requirements that did move sit one layer down.

The fetch became the product

Getting a page is no longer a function call. Against a defended target it is a negotiation across TLS fingerprints, HTTP/2 frame ordering, header casing and order, IP reputation, JavaScript challenges, behavioural signals and session continuity. The defences change without notice and without a changelog. Our guide to why scrapers get blocked walks the individual signals; the property that matters here is that an adversary maintains them.

No library can vendor that. A framework maintainer cannot ship a fingerprint that stays correct, because correctness is defined by a third party who updates on their own schedule. The limit is structural. Any component whose correctness decays on someone else's timetable has to be either continuously maintained by a dedicated team or continuously bought.

So the frameworks did the only sensible thing available to them. They made the blocking layer a seam instead of a feature.

Crawlee's proxy documentation is explicit. Its ProxyConfiguration exists so that, in the docs' own words, "we can use our own proxy servers or proxy servers acquired from third-party providers". There is a newUrlFunction hook for custom selection logic and a sessionId parameter that pins a session to a stable proxy, so repeated requests read as one visitor rather than a crowd. Scrapy solves the same problem through downloader middlewares, which is precisely why a release about SOCKS support and TLS version pinning counts as a substantial release. Both projects treat the network layer as something you supply.

That seam is the design decision that keeps these projects usable. Without it they would have gone stale years ago.

The second adaptation: output shape

The other thing that changed is what comes out the far end. Downstream demand moved toward model input, and a wall of DOM nodes is poor model input.

Crawl4AI is built around that shift. Its documentation describes it as "blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data pipelines", and the first capability it lists is to "Generate Clean Markdown: Perfect for RAG pipelines or direct ingestion into LLMs". It also releases quickly, with 0.9.0 on 18 June 2026, 0.9.1 on 8 July and 0.9.2 on 15 July, which is what a project looks like when its problem is still in motion. Whether the model or a selector should do the extraction is a separate decision, and our comparison of the two approaches treats it field by field.

Katana shows the same adaptation from the security reconnaissance side. Its v1.6.0 release on 4 May 2026 added automatic login, per-domain page budgets, adaptive rate limiting and page load strategy options, with v1.6.1 following the next day. Automatic login and adaptive rate limiting are fetch-layer concerns. Per-domain budgets are a crawl-layer concern. A crawler that survives has to keep both in view.

The projects that look healthiest in mid-2026 have two things in common: a pluggable blocking layer and an output format a model can consume. Neither has much to do with crawling.

Where the build-versus-buy line runs

Building a crawler is cheap and always was. Pick something from the crawling frameworks category, write the spiders, run them. Playwright covers the rendering cases plain HTTP cannot reach. That is not where the recurring cost lives.

Buying the fetch is where the money and the risk sit. That is what a managed scraping API or a proxy network sells: someone else's full-time attention to a moving target. Zyte, staffed by people who have worked on Scrapy for years, and Apify, which maintains Crawlee, both sell exactly this. That is not a contradiction. Both give away the solved layer and charge for the unsolved one, because that is where their maintenance goes.

The architecture that follows is the one most teams should run: an open-source crawler you control, a paid fetch backend behind an interface you own, and nothing in the first that assumes anything about the second. We sketched the three-layer version of this in an earlier post on the 2026 stack.

The interface is the hedge

The fetch layer is also the least stable business layer in this market, which is the practical reason to keep it swappable.

Proxyway's news index, read on 27 July 2026, carries three items from a single month that all land on this layer: "Oxylabs Receives $130 Million Investment from Warburg Pincus" dated 10 July 2026, "SOAX Changes Its Web, Products, Pricing" dated 7 July 2026, and "The FBI & Google Disrupt NetNut's Residential Proxy Network" dated 3 July 2026. Those are Proxyway's headlines and dates as published. The underlying details are theirs rather than ours, and the last one in particular describes an action whose outcome is not settled. Taken together they describe an ordinary month on this layer, with capital arriving and prices moving while one supplier's network was being pulled apart in public. We looked at the capital side of that in more depth in the proxy consolidation post.

If your crawl logic imports a vendor SDK directly, each of those events is a code change. If it calls a fetch(url) interface you defined, each one is a config change. The interface takes about a day to write and saves you the rewrite every time this layer moves.

What this means for a team collecting web data

Do not pay a managed API for the crawling. Scheduling, dedup, retries and parsing are free and stable, and a metered endpoint bills you per page for logic you could run yourself.

Do pay for the fetch on targets that fight back. That is the maintained part. If nobody on your team will own fingerprint drift week after week, buy it, and read the multi-vendor failover guide before signing with one supplier.

Put an interface between the two. One function, your signature, vendor code only on the far side of it. Everything above should be testable against a local fixture with no vendor account present.

Split targets by fetch difficulty rather than by tool. Cooperative targets run on a bare framework. Defended ones route through a paid backend. Same crawler, different backend, decided per target rather than per project.

It is also worth logging block rate per target and per backend before you need the numbers. When a vendor degrades or reprices, you want to already know what it was doing for you.

If you are choosing a framework at all, the choice is mostly about language and team familiarity. Scrapy versus Crawlee is a real comparison, but it rarely decides whether your crawl works. That gets decided by whichever anti-bot system stands between your crawler and the page, and by what you are willing to pay to get past it.