How to scrape multiple pages (pagination and infinite scroll)
In short: Scrape multiple pages by first identifying how the site splits its results: numbered pagination with page URLs, a "Load more" button, or infinite scroll. Numbered pages can be visited one by one and merged; "Load more" buttons and infinite scroll need a tool that clicks or scrolls the rendered page until every batch has loaded.
First, identify the pagination type
Scroll to the bottom of the listing and look at what the site does when there is more to show:
- Numbered pagination: page links like 1 · 2 · 3 · Next, usually with a URL that changes (
?page=2,/p/3). - "Load more" button: a button that adds items to the current page without changing the URL.
- Infinite scroll: new items appear automatically as you scroll down; there is no button and no page number.
The type dictates the method. Get this right and the rest is straightforward.
Numbered pagination
This is the friendliest case because the page number lives in the URL. Two ways to handle it:
- Follow the pattern. If page two is
?page=2, you can walk through?page=1,?page=2, and so on until the results run out. A tool can do this automatically by detecting and clicking the "Next" link. - Extract each and merge. Pull each page and stack the rows into one file, deduplicating any overlap.
The key is that every page is a real URL, so nothing is hidden — you just need to visit them all and combine the results.
"Load more" buttons
Here the URL never changes; the button injects more items into the same page with JavaScript. A tool that reads the raw HTML once will only ever see the first batch. What you need is something that clicks the button repeatedly on the rendered page, waits for each batch to load, and only then collects everything. Manually, that means clicking "Load more" until it disappears, then extracting — tedious, but it works.
Infinite scroll
Infinite scroll is the same idea as "Load more" but triggered by scrolling rather than clicking. As you reach the bottom, JavaScript fetches the next batch and appends it. Because the full list never exists in the initial HTML, you have to keep scrolling to force everything to load before collecting. Some sites also remove off-screen items as you go to save memory, which makes manual collection genuinely hard — you can lose the top of the list while loading the bottom.
The easy way — let the tool do the walking
Handling clicks, scrolls, waiting, and merging by hand is exactly the sort of repetitive work worth automating. With Pro, ScrapeSheet follows the pages for you and gathers the whole set into one export. The free plan allows up to 20 rows per export and one page per run; ScrapeSheet Pro (€12.99 per month) removes both limits.
- Install ScrapeSheet from the Chrome Web Store and open the listing page.
- Click the icon and choose Extract.
- With Pro, it follows the pagination or loads the infinite-scroll list for you, collecting every result.
- When it finishes, export the combined result to Excel (
.xlsx) or CSV — one clean file, all pages.
Because it works on the page as your browser displays it, it can collect infinite-scroll lists that formula-based approaches like Google Sheets' IMPORTHTML cannot see.
Do it responsibly
Collecting many pages means many requests. Scrape at a reasonable, human-like pace so you do not strain the server; respect the site's rules, including its robots.txt file (the format is standardized in RFC 9309); and take only the pages you actually need. More on this in is web scraping legal? Once you have the full list, the export step is identical to any single page — see how to scrape a website to Excel.
More step-by-step guides are on the ScrapeSheet blog.
FAQ
How do I scrape data across multiple pages?
Identify the pagination type first. For numbered pages, you can loop through the URL pattern; for "Load more" buttons or infinite scroll you need a tool that clicks and scrolls the rendered page. ScrapeSheet Pro follows pagination and loads infinite-scroll lists, merging the results into one export.
What is infinite scroll and why is it hard to scrape?
Infinite scroll loads more items with JavaScript as you scroll down, so the full list never exists in the initial HTML. You have to keep scrolling to trigger loading before you can collect everything.
How many pages can I scrape?
With ScrapeSheet, the free plan covers one page per run and Pro follows all pages. Whatever the tool, scrape at a reasonable pace, respect the site's rules, and take only the pages you need.
Sources
General information, not legal advice. Always check the website's terms and the data-protection rules that apply to you.