Pagination Handling

Pagination handling is how a crawler walks a result set that a site serves in slices. Four patterns cover nearly everything, and each fails in its own way.

Offset or page-number URLs, the ?page=3 or ?offset=60 form, are trivially parallelizable: you can request every page at once because the addresses are predictable. They also drift. The underlying list is usually a live query, so an item inserted between your request for page 2 and page 3 pushes one record across the boundary, and you get it twice while another is never returned. Sorting by a stable key such as an id or creation date instead of by relevance or price removes most of that. Cursor or token pagination is the opposite trade: each response carries the pointer to the next, so the sequence is consistent under inserts, but you cannot request page 40 without walking pages 1 through 39. Throughput per result set is capped at one request round trip at a time, and parallelism has to come from running many result sets side by side.

Infinite scroll and load-more buttons are the same thing with different chrome. Both fetch more rows through XHR or fetch as you interact. You can drive them with a browser, scrolling in Playwright or clicking through Crawlee, and pay for a browser process per result set. The far better move is to open the network panel, find the endpoint the page itself calls, and request that JSON directly. It usually accepts a limit and offset, returns clean typed records with no parsing, and costs a plain HTTP fetch. Point-and-click tools such as Octoparse and Browse AI wrap the browser-driven version for teams that would rather not maintain the endpoint contract themselves.

Result-set caps are the trap that makes a partial harvest look finished. Many sites refuse to paginate past a fixed depth, commonly around 1,000 results, returning an empty page or silently repeating the last one. If a category claims 40,000 listings and the crawler stops cleanly at page 50, that is a cap, not the end of the data. The fix is query partitioning: slice by facet, price band, geography or date range until each slice returns fewer results than the cap, then union the slices and deduplicate. Two checks are worth building in from the start. Ask whether the underlying endpoint can be called directly, because that decision usually dominates crawl cost. Then compare the count you harvested against the count the site advertises, per partition, and treat any round-numbered stopping point as a cap until proven otherwise.