Web Scraping

Web scraping is the practice of retrieving web pages programmatically and extracting structured records from them. The output is a table, not a page: prices with currencies, listings with dates, profiles with fields. Two neighboring activities get confused with it. Crawling is the discovery half, walking links to decide which URLs exist and which are worth fetching, and a crawler that never extracts anything is still a crawler. Consuming an API is retrieval of data a publisher has already chosen to expose in machine form, with a contract, a rate limit and usually a key. Scraping is what you do when no such contract exists and the data only ships as a rendered document.

Every scraping system has four stages, whether or not the team ever named them. Fetch gets bytes over the network and fails as blocks, challenges, timeouts and rate limits; its cost scales with how defended the target is. Render executes JavaScript so client-built content exists at all, and fails as timeouts, missing hydration and memory exhaustion; it is usually the most expensive stage per page by a wide margin, because it means running a browser. Parse turns a response into typed records and fails silently, which is the dangerous part, since a broken selector returns empty fields rather than an error. Store handles deduplication, schema changes and history, and fails as duplicates, gaps and records nobody can reconcile against last week's run.

The build-versus-buy line usually falls between the second and third stages. Fetch and render are commodity problems with brutal fixed costs, so most teams end up buying them from a scraping API such as ScraperAPI, Firecrawl or Apify. Parse and store stay in-house, because the schema is specific to the business and nobody outside it knows what a correct record looks like. Frameworks like Scrapy sit on the other side of that line, giving you the scheduling and pipeline scaffolding while you supply proxies yourself.

The policy frame matters at three separate levels, and they are not one instrument. RFC 9309 says of robots.txt rules that "these rules are not a form of access authorization" and that the protocol "is not a substitute for valid content security measures", so treat it as a crawling convention rather than a legal boundary. Terms of service are contract, enforced differently again. Personal data carries obligations under GDPR, CCPA and their peers regardless of how you obtained it, including data scraped from public pages. Jurisdiction decides which of these applies. Get counsel for the specific target rather than reasoning from a headline case.