Is web scraping legal?
In short: Is web scraping legal? Scraping is generally not illegal in itself, but it depends on what you collect, how you access it, and what you do with it. The site's terms, robots.txt, personal data under laws such as the GDPR, and copyright drive most of the risk. This is general information, not legal advice.
Not legal advice. This article is general information only. Laws differ by country and change over time, and every situation is different. For anything important, or anything commercial, talk to a qualified lawyer in your jurisdiction.
The short version
Reading a public web page is something billions of browsers and search engines do every day, and collecting publicly available information is, in many contexts, permitted. Scraping is generally not illegal in itself. What can create problems is the combination of what you collect, how you access it, and what you then do with it. Three things drive most of the risk: the site's rules, the nature of the data, and your conduct.
Robots.txt — the polite convention
Most sites publish a file at /robots.txt that tells automated crawlers which paths the site would prefer bots not to visit. Its format is standardized in RFC 9309. It is a widely respected convention rather than a law in most places, but honoring it is good practice and a clear signal of good faith. If a section is disallowed, treat that as a request to stay out. You can read any site's file by adding /robots.txt to its domain.
Terms of service
A site's terms of use may restrict automated access or bulk collection. Terms can be legally relevant — especially where you have an account and have agreed to them, or where access is clearly conditioned on them. They are not identical everywhere, and their weight varies, but ignoring an explicit prohibition is a risk you should take seriously. Read the terms before scraping a site systematically, particularly behind a login.
Personal data and the GDPR
This is the area that most often turns a harmless-looking scrape into a real problem. Names, email addresses, phone numbers, photos, and profiles are personal data, and their collection and use are regulated by laws such as the EU GDPR, the UK GDPR, and similar frameworks elsewhere. "It was public" does not automatically make it fair game: you generally still need a lawful basis to process it, and duties around transparency, purpose, and retention can apply. When scraping, the basis usually relied on is legitimate interests (Article 6(1)(f)), which requires weighing your purpose against the rights and reasonable expectations of the people concerned. Those people keep their rights, including the right to object (Article 21), which must always be honored for direct marketing. Again, this is general information, not legal advice.
- Prefer non-personal data (prices, specs, availability, aggregate figures) whenever you can.
- Do not build marketing lists of people's contact details from scraped pages without proper legal grounds and consent where required.
- If you must handle personal data, minimize what you keep, secure it, and be ready to justify why you have it.
Copyright and databases
Facts are generally not owned by anyone, but the way they are expressed can be — articles, photos, and creative text are usually protected by copyright, and in the EU some databases carry their own sui generis protection. Copying and republishing substantial protected content is a different matter from noting a price or a spec. Use scraped content for analysis and internal decisions rather than wholesale republication.
How to scrape responsibly
- Check robots.txt and the terms first. If something is clearly off-limits, respect it, and never get around logins, CAPTCHAs, or other access controls.
- Be gentle. Do not flood a server with rapid-fire requests; scrape at a human-like pace so you do not degrade the site for others.
- Take only what you need. Favor non-personal, factual data and a narrow scope.
- Use it responsibly. Analysis, price comparison, and research sit very differently from republishing content or profiling people.
- When in doubt, ask. Many sites offer an official API or will grant permission if you ask.
Where a tool like ScrapeSheet fits
ScrapeSheet works on the page already open in your own browser and exports what you see to Excel or CSV — it is closer to an assisted copy-paste than an aggressive automated crawler hitting a server thousands of times. Even when Pro follows many pages, keep the pace gentle and the scope narrow. ScrapeSheet does not, of course, decide the legality of any particular use — that is on you, and the responsible habits above still apply. You can see the everyday jobs it is built for on the use cases page.
For the practical side, read how to scrape a website to Excel and the other guides on the ScrapeSheet blog.
FAQ
Is web scraping illegal?
Scraping is generally not illegal in itself, and accessing publicly available data is often permitted. But what you scrape, how you access it, and what you do with it can raise issues under terms of service, copyright, and data-protection law. This is general information, not legal advice.
What does robots.txt do?
Robots.txt is a file where a site states which paths automated crawlers should not access. It is a widely respected convention rather than a law in most places, and honoring it is good practice.
Can I scrape personal data?
Personal data (names, emails, profiles) is regulated by laws such as the GDPR. Collecting or processing it needs a lawful basis (for scraping, usually legitimate interests, which requires a balancing test) and respect for people's rights, such as the right to object; when in doubt, avoid it or seek professional advice.
Sources
- Regulation (EU) 2016/679, General Data Protection Regulation (EUR-Lex)
- RFC 9309: Robots Exclusion Protocol (RFC Editor, IETF)
General information, not legal advice. Always check the website's terms and the data-protection rules that apply to you.