How to scrape multiple pages (pagination and infinite scroll)
The first page is easy. The problem is that your data almost never fits on it — a search result, a product catalogue or a directory runs across dozens of pages, and you want all of it in one spreadsheet. The trick is to recognise how the site splits its content, because pagination, "Load more" buttons and infinite scroll each need a different handling.
First, identify the pagination type
Scroll to the bottom of the listing and look at what the site does when there is more to show:
- Numbered pagination: page links like 1 · 2 · 3 · Next, usually with a URL that changes (
?page=2,/p/3). - "Load more" button: a button that adds items to the current page without changing the URL.
- Infinite scroll: new items appear automatically as you scroll down; there is no button and no page number.
The type dictates the method. Get this right and the rest is straightforward.
Numbered pagination
This is the friendliest case because the page number lives in the URL. Two ways to handle it:
- Follow the pattern. If page two is
?page=2, you can walk?page=1,?page=2and so on until the results run out. A tool can do this automatically by detecting and clicking the "Next" link. - Extract each and merge. Pull each page and stack the rows into one file, de-duplicating any overlap.
The key is that every page is a real URL, so nothing is hidden — you just need to visit them all and combine the results.
"Load more" buttons
Here the URL never changes; the button injects more items into the same page with JavaScript. A tool that reads the raw HTML once will only ever see the first batch. What you need is something that clicks the button repeatedly on the rendered page, waits for each batch to load, and only then collects everything. Manually, that means clicking "Load more" until it disappears, then extracting — tedious, but it works.
Infinite scroll
Infinite scroll is the same idea as "Load more" but triggered by scrolling rather than clicking. As you reach the bottom, JavaScript fetches the next batch and appends it. Because the full list never exists in the initial HTML, you have to keep scrolling to force everything to load before collecting. Some sites also remove off-screen items as you go to save memory, which makes manual collection genuinely hard — you can lose the top of the list while loading the bottom.
The easy way — let the tool do the walking
Handling clicks, scrolls, waiting and merging by hand is exactly the sort of repetitive work worth automating. ScrapeSheet detects the pagination type for you and gathers the whole set into one export.
- Install ScrapeSheet from the Chrome Web Store and open the listing page.
- Click the icon and choose Extract.
- Turn on following pages: it clicks "Next", presses "Load more", or scrolls an infinite feed as needed, loading every batch.
- When it finishes, export the combined result to Excel (
.xlsx) or CSV — one clean file, all pages.
Because it works on the rendered page, it copes with the JavaScript-driven loading that defeats formula-based approaches like Google Sheets' IMPORTHTML.
Do it responsibly
Collecting many pages means many requests. Scrape at a reasonable, human-like pace so you do not strain the server, respect the site's rules, and take only the pages you actually need. More on this in is web scraping legal? Once you have the full list, the export step is identical to any single page — see how to scrape a website to Excel.
FAQ
How do I scrape data across multiple pages?
Identify the pagination type first. Loop the URL for numbered pages; for "Load more" or infinite scroll use a tool that clicks and scrolls the rendered page. ScrapeSheet follows all three and merges the results.
What is infinite scroll and why is it hard?
It loads more items with JavaScript as you scroll, so the full list is never in the initial HTML — you must keep scrolling to trigger loading before collecting.
How many pages can I scrape?
As many as the site serves, but keep a reasonable pace, respect the site's rules, and take only what you need.