When a No-Code Scraper Stops Working: Selector Drift and the Move to Code

Browse AI's own pricing example says that pulling 50 products off a listing page costs 5 credits, and then opening each of those 50 detail pages costs 50 more. The detail pages cost ten times what the listing did, on the same job, documented on the same page. That kind of thing never shows up in the demo, and the demo is not lying either: you install the extension, click three prices, hit run, and twenty seconds later there is a CSV. Month four is the part nobody writes up. Tuesday's run came back with 200 rows instead of 4,000, the price column is empty on half of them, and the invoice says $340.

Vendors have obvious reasons not to cover this. Most review sites just do not. Proxyway's reviews of Octoparse and ParseHub score features against a rubric and land on a number, which tells you what a tool claims to do and nothing about how it stops working. So: five limits that actually end a no-code build, what each looks like from the inside, and how to plan the handoff to code before a downstream job breaks.

I am not arguing against no-code scrapers here. They are the fastest path from a question to a spreadsheet, and for plenty of jobs they are the correct final answer. The point is knowing which job you have.

Limit one: selector drift

When you click an element in a recorder, the tool derives a path to it. Web Scraper's documentation describes selectors organised in a tree that the scraper walks in order, which is a clean model and also exactly the thing that breaks. The stored path might be a CSS selector, an XPath, or a positional index. None of those carry meaning. Nothing in the recording says "this is the price." It says "this is the second span inside the div with class _1f8xk2."

Generated class names are the usual trigger. A team ships a build with a new version of their CSS tooling, every hashed class changes, and every scraper pointed at that site loses every field at once. Nothing about the page looks different to a human. Rendering changes do the same damage more quietly: a block that used to arrive in the HTML now arrives from a client-side fetch, and an HTTP-based scraper sees an empty container where the data used to be. The worst case is an A/B test, which breaks the scraper on some runs and not others, so it reads like a flaky network rather than a broken selector.

The failure mode matters more than the cause. A broken selector almost never throws. It returns nothing, the run completes, the export is written, and the row count drops. If nobody is watching row counts, the first person to notice is whoever consumes the data three weeks later.

Two practical defences work regardless of which tool you use. Alert on row count deltas rather than on job status, because job status will be green. And prefer selectors anchored to stable attributes over positional ones: an id, a data-testid, a [itemprop] attribute from schema.org markup, or an aria-label will all outlive a class-name refactor. Most recorders let you edit the selector by hand after recording it. Almost nobody does.

Limit two: pagination and infinite scroll that only worked on page one

Every no-code tool on this site advertises pagination and infinite scroll, and the claims are honest for the common cases. Octoparse documents handling of JavaScript-rendered pages, infinite scroll and pagination; ParseHub documents forms, dropdowns, infinite scroll and pagination; Web Scraper builds pagination into its sitemap model and even ships a selector that pulls URLs straight out of a site's sitemap.xml so you can skip navigation entirely. For a numbered pager or a next-page link, these work.

The variants are where recorded workflows fall apart:

Cursor-based feeds. The next page is a token in a request, not a URL. There is no "next" link to click, so a recorder has nothing to record. Some tools can scroll their way through it, which works until the feed is long enough to exhaust memory or time out.

Virtualised lists. The page removes rows from the DOM as you scroll past them to keep the browser fast. The scraper scrolls to the bottom, reads the DOM, and finds the last 40 rows of a 4,000-row list. The run succeeds.

Template switches mid-crawl. Pages one to ten use one layout, page eleven switches to a different one because it was built by a different team, and the selector tree silently stops matching.

Termination conditions. A recorded loop needs to know when to stop. If the load-more button disappears at the end, fine. If it stays and does nothing, the loop runs until a timeout, and on usage-based pricing that timeout has a cost attached.

The underlying issue is that recording captures one traversal of one session. Code can express a rule: keep going while the response contains a cursor, stop after N pages, stop when a page yields zero new IDs. A recorder can only replay what it saw.

Limit three: per-run pricing that stops being cheap

No-code tools are cheap at small volume and stop being cheap at a threshold that depends entirely on the billing unit. Comparing plan prices across vendors is close to meaningless, because they meter different things.

Figures as listed on each vendor's own pricing page, read on 27 July 2026. Treat them as shapes rather than quotes. Octoparse listed Standard at $69 per month and Professional at $249 per month, but both of those are the annual-billing rates with a 16 percent discount already applied, so paying month to month costs more than the headline. That gap between the advertised number and the number you pay is worth checking on every vendor in this table before you model anything. Apify listed Free at $0, Starter at $29, Scale at $199 and Business at $999, with platform usage billed on top of the subscription. Web Scraper listed Project at $50 with 5,000 URL credits, Professional at $100 with 20,000, and Scale from $200 with unlimited credits. ParseHub's pricing page did not return readable pricing when checked on the same date, so I am not quoting a ParseHub figure here; get it from the vendor directly. Confirm everything at signup.

The comparison that decides the migration is not vendor against vendor. It is the monthly no-code bill against the fully loaded cost of running the same crawl on your own infrastructure, which is a framework you do not pay for plus proxy bandwidth you do. That second number is not small and it is not zero-maintenance. Our guide to proxy pricing models covers how to normalise per-GB, per-IP and per-request quotes into one comparable figure, which you will need before the comparison means anything.

A rough heuristic that survives contact with reality: if the no-code bill is under roughly a day of engineer time per month, the tool is winning. If it is over a week of engineer time per month, the tool is losing and you are paying for convenience you have already outgrown. Between those, look at the other four limits and decide on those instead.

Limit four: logins, rate limits and everything a recorder cannot express

A recorder captures a sequence of actions. It does not capture conditions, and the difference shows up fast on any target with authentication or an aggressive rate limiter.

Login is the obvious case. Recording a form fill works. What a recording cannot express is: if the session cookie is older than 20 minutes, refresh it; if the response is a login redirect, re-authenticate and retry the original request; if the account hits a device check, stop and page someone. Some tools handle a slice of this. Octoparse documents login flows and CAPTCHA automation, and Browse AI documents CAPTCHA solving. Those cover the recorded happy path. They do not give you a place to put an if.

Rate limiting is the subtler case. A well-behaved crawler reads a 429 and backs off, ideally honouring Retry-After, and ramps concurrency back up once responses recover. Scrapy ships AUTOTHROTTLE and RETRY_TIMES for exactly this, and Crawlee has an autoscaled pool that adjusts concurrency against system load and error rates. A recorded workflow has one speed setting and one retry count. Under a rate limiter, that gets you either a crawl that is far slower than it needs to be or a burst that gets the whole account blocked.

Then there is the wider blocking problem. Cloud-run no-code tools bundle proxies and challenge handling, which is a genuine advantage over running the same job from your own laptop. But the bundle is opaque. When a target starts returning challenge pages, you cannot see which layer failed, cannot pin a session to one IP, and cannot swap the proxy type. Our guide on why scrapers get blocked walks the diagnostic order, and choosing a proxy type by target covers the decision you will eventually have to make yourself. Both assume a level of control that a bundled no-code runner does not give you.

Who this is wrong for: if your targets are cooperative public listings with no auth and no meaningful anti-bot system, none of this section applies to you and you should ignore it.

Limit five: output schemas that drift silently

This one causes the most damage per incident, because it fails downstream rather than at the scraper.

A no-code export is a CSV or a JSON blob shaped by whatever the run happened to produce. There is no schema, no type contract and no validation. A field that disappears becomes an absent column. A price that switches from $1,299.00 to 1299.00 USD becomes a string your parser silently coerces to zero. A field that returns a list on some pages and a scalar on others produces a file that half your downstream jobs can read.

If a human opens the CSV, this is a minor annoyance. If the export feeds a dashboard, a pricing model or a database load, it is a data quality incident that is discovered late, because the load succeeded.

Code does not fix this for free, but it gives you the place to fix it. Scrapy's Item and ItemLoader classes with an item pipeline let you assert that a price is a decimal and a URL is absolute, and drop or quarantine anything that fails. Crawlee's dataset writes are ordinary function calls, so you validate before pushData with whatever library you already use. In both cases the check lives in version control, next to the code that produces the data, and it fails loudly.

The interim measure, if you are staying on no-code for now: validate on the receiving side. A ten-line script that checks row counts, null rates per column and type consistency against the previous run will catch most of these before anyone else does.

Does AI field detection fix selector drift?

Partly, and the two vendors making the loudest claims are not claiming the same thing.

Browse AI's features page, read on 27 July 2026, says its robots "will adapt to layout changes, and you don't need to re-train your robot for UI changes." That is a runtime claim: the robot is meant to keep working through a redesign. Octoparse makes a different one. Its site invites you to "Let AI-powered Auto-detect draft your website workflow, then customize with simple drag-and-drop," and says the tool will retrieve the important parts of a page without human intervention. That is a build-time claim: the tool guesses the structure when you create the task, and rebuilding after a redesign is faster because you re-run auto-detect rather than re-clicking every field. Both are useful. They are not the same feature, and a review that scores both as "AI extraction" is flattening a real distinction. I did not find equivalent self-healing claims on ParseHub's or Web Scraper's own sites on the same date, which is not the same as confirming they have none.

Neither vendor publishes a measured accuracy rate for these features, and no independent benchmark of self-healing extraction that I could verify exists. So the honest position is: AI field detection shortens the rebuild and probably reduces how often you need one. It does not tell you when it has guessed wrong, which is the property you would actually need for it to replace monitoring. A robot that adapts to a redesign by confidently selecting the wrong element produces a full CSV of bad data, which is worse than an empty one.

The same caveat applies to the broader AI extraction category, where a model reads the page and returns structured fields. It moves the fragility from a CSS selector to a prompt and a model version. That is often a good trade. It is not the absence of a trade, and it comes with a per-page inference cost that behaves like the credit systems above.

What to port first

The migration mistake is treating it as all or nothing. You do not rewrite the account. You rewrite one target.

Pick the target with the worst ratio of maintenance to value. Usually that is the highest-volume one, the one that broke most recently, or the one a downstream job depends on. Port that. Leave everything else where it is.

Then run both for a week. The old robot and the new code hit the same target on the same schedule, and you diff the outputs daily. This catches the things a careful reading of the recorded config will not: a filter you forgot was applied, a field that was being cleaned by the no-code tool's built-in parser, a deduplication rule that lived in the platform rather than in your head. A week of parallel running costs one week of double credits and saves the argument about whether the new pipeline is correct.

Only then turn the robot off.

Mapping a recorded workflow onto Scrapy or Crawlee

The recorded config is a specification. Read it as one. Web Scraper's sitemaps are JSON and support export and import, so the selector tree is already machine-readable and you can work from the file rather than from screenshots of the UI.

Which framework depends on the language your team already writes. Scrapy is the mature Python option with the deepest ecosystem for structured pipelines and large crawls, and it does no anti-bot work of its own, so hard targets need a proxy network underneath it. Crawlee ships request queuing, session pooling and anti-blocking helpers out of the box, its Node story is more mature than its Python port, and its adaptive crawler is documented in both. The Scrapy vs Crawlee comparison goes into the differences properly.

Where a recorded workflow used a real browser and you need to keep that, Playwright is the direct equivalent: it drives Chromium, Firefox and WebKit through one API with auto-waiting and network interception, which is what the no-code tool was doing on your behalf. It is also the biggest cost jump, because a browser per page is expensive in a way that the credit system was hiding from you. Run browsers only for the pages that need them, which is exactly what Crawlee's adaptive crawler is for.

Who this migration is wrong for: a team with no Python or Node engineer. If nobody on the team can maintain the code after the person who wrote it leaves, a no-code tool with an annoying bill is a better outcome than an unmaintained repository. That is a real answer, not a consolation prize.

The middle option: keep the tool, move the fetch loop

There is a third position that gets ignored because neither side of the argument sells it.

Keep the no-code tool for discovery. Point-and-click is genuinely the fastest way to work out what a page contains, which fields exist, how the pager behaves and whether the data is worth collecting at all. That is a prototyping strength, and it does not expire when you start writing code. Use the recorder to answer "is there anything here," then throw the robot away and write the ten lines that run on a schedule.

Or split by target. High-volume, high-value, breaks-often targets go into code. The long tail of ten sites you check monthly stays in the recorder, where the maintenance cost is a few minutes of re-clicking. This is the setup most teams end up with, and it is stable rather than transitional.

Or split by layer. Apify is the clearest case here, because it sits on both sides of the line: a marketplace of prebuilt actors you can run without writing anything, and a platform for running actors you wrote yourself with Crawlee, which Apify maintains. Moving from "someone else's actor" to "our actor" on the same platform is a much smaller step than moving off the platform entirely, and it keeps the scheduling, storage and proxy layers you have already integrated. The trade is that the hosted platform is closed source and not self-hostable, so you have solved the code-ownership problem and not the vendor-dependency one.

Which of those you pick depends on where your pain actually is. If it is maintenance, code the fetch loop. If it is cost, look at the billing unit first, because switching from a per-page to a per-compute model sometimes fixes it without any migration at all. If it is capability, the recorder was never going to get there and no plan upgrade will change that.

Where this leaves the decision

The ceiling has nothing to do with tool quality. These are competent products built by people who know the domain, and the demo works because the demo is honest about the case it shows. The limit is that a recorder can only store the path it was shown, replay the session it watched, and hand you whatever file that produced. Intent, rules and contracts live somewhere else.

Any one of the five limits above is survivable on its own. Builds end when two or three land in the same month on a target something else depends on.

The insurance is cheap. Track row counts week to week and drift shows up as a graph instead of a complaint from someone downstream. Learn your billing unit once and the invoice stops surprising you. Decide now which single target you would port first, because deciding that under pressure is how teams end up rewriting everything at once. Do those and the migration becomes a sprint item you schedule, and the recorder keeps doing the work it was always good at.

For the wider build-versus-buy picture, the web scraping stack in 2026 covers how the three layers fit together, and how to choose a web scraping tool works through the category decision from the top. If you have not picked a no-code tool yet, deployment model is the choice that constrains everything else, including how hard this migration will be when it comes.