DOM (Document Object Model)

The DOM is the in-memory tree a browser builds from an HTML document and then mutates as scripts run. It is an object graph with an API, not a text format. The distinction that matters most in scraping follows directly: the HTML the server sent is not the DOM the user sees. On a modern site the two can share almost no structure.

A scraper can target three states. The raw response bytes are what an HTTP client receives, before any script has run. The DOM after initial parse is that markup as a tree, with malformed regions repaired and implied elements inserted. The DOM after script execution and hydration settle is what DevTools shows you. Selectors are almost always written against the third state and then run against the first, which is why a selector you verified by right-clicking in the browser fails on the first real request and returns nothing. Nothing errored. The node simply did not exist yet.

Two structures break otherwise correct selectors without any obvious symptom. Shadow DOM encapsulates a component's internal tree behind a boundary that ordinary document-level queries do not cross, so a selector matching a node you can see in the inspector matches nothing from script. Tooling differs here: the Playwright documentation states that "CSS selectors pierce open shadow DOM", while noting separately that "XPath does not pierce shadow roots", so switching selector dialect silently changes what is reachable. Iframes are a harder boundary again, since the embedded document is a separate document, and any content inside one must be addressed through a frame handle rather than found in the parent.

Before committing to browser rendering, run the cheap test: fetch the raw HTML with curl or a plain HTTP client and search it for a value you need. A surprising share of client-rendered pages ship their records in the initial payload anyway, in a JSON blob assigned to a global variable, in a Next.js __NEXT_DATA__ script tag, or in an embedded JSON-LD block. If the data is there, skip the browser and extract from JSON, which is both more stable than selectors and far cheaper. That call is usually the largest single cost difference in a crawl: a plain fetch is a few kilobytes and milliseconds of CPU, while a rendered page in Playwright or Puppeteer is a browser process, hundreds of megabytes of memory and seconds of wall clock. Crawl4AI and similar tools let you try the cheap path first and fall back only where the raw fetch comes up empty. Render by exception, not by policy.