Is web scraping legal?
It is one of the most common questions people ask before their first export — and the honest answer is: it depends. Scraping is a tool, and like any tool it can be used well or badly. This guide explains the main things that matter — robots.txt, terms of service, personal data and copyright — so you can make sensible decisions.
The short version
Reading a public web page is something billions of browsers and search engines do every day, and collecting publicly available information is, in many contexts, permitted. Scraping is generally not illegal in itself. What can create problems is the combination of what you collect, how you access it, and what you then do with it. Three things drive most of the risk: the site's rules, the nature of the data, and your conduct.
Robots.txt — the polite convention
Most sites publish a file at /robots.txt that tells automated crawlers which paths they would prefer bots not to visit. It is a widely respected convention rather than a law in most places, but honoring it is good practice and a clear signal of good faith. If a section is disallowed, treat that as a request to stay out. You can read any site's file by adding /robots.txt to its domain.
Terms of service
A site's terms of use may restrict automated access or bulk collection. Terms can be legally relevant — especially where you have an account and agreed to them, or where access is clearly conditioned on them. They are not identical everywhere, and their weight varies, but ignoring an explicit prohibition is a risk you should take seriously. Read the terms before scraping a site systematically, particularly behind a login.
Personal data and the GDPR
This is the area that most often turns a harmless-looking scrape into a real problem. Names, email addresses, phone numbers, photos and profiles are personal data, and their collection and use are regulated by laws such as the EU/UK GDPR and similar frameworks elsewhere. "It was public" does not automatically make it fair game: you generally still need a lawful basis to process it, and duties around transparency, purpose and retention can apply.
- Prefer non-personal data (prices, specs, availability, aggregate figures) whenever you can.
- Do not build marketing lists of people's contact details from scraped pages without proper legal grounds and consent where required.
- If you must handle personal data, minimise what you keep, secure it, and be ready to justify why you have it.
Copyright and databases
Facts are generally not owned by anyone, but the way they are expressed can be — articles, photos, and creative text are usually protected by copyright, and in the EU some databases carry their own sui generis protection. Copying and republishing substantial protected content is a different matter from noting a price or a spec. Use scraped content for analysis and internal decisions rather than wholesale republication.
How to scrape responsibly
- Check robots.txt and the terms first. If something is clearly off-limits, respect it.
- Be gentle. Do not flood a server with rapid-fire requests; scrape at a human-like pace so you do not degrade the site for others.
- Take only what you need. Favour non-personal, factual data and a narrow scope.
- Use it responsibly. Analysis, price comparison and research sit very differently from republishing content or profiling people.
- When in doubt, ask. Many sites offer an official API or will grant permission if you ask.
Where a tool like ScrapeSheet fits
ScrapeSheet works on the page already open in your own browser and exports what you see to Excel or CSV — it is closer to an assisted copy-paste than an aggressive automated crawler hitting a server thousands of times. That design nudges you towards the gentle, small-scale end of the spectrum, which is exactly where most everyday, low-risk scraping lives. It does not, of course, decide the legality of any particular use — that is on you, and the responsible habits above still apply.
FAQ
Is web scraping illegal?
Not in itself, and accessing public data is often permitted. Risk comes from what you scrape, how you access it, and what you do with it, under terms of service, copyright and data-protection law. General information, not legal advice.
What does robots.txt do?
It states which paths a site prefers crawlers not to access — a respected convention rather than a law, and worth honoring.
Can I scrape personal data?
Personal data is regulated by laws like the GDPR. Collecting or processing it needs a lawful basis and care; when unsure, avoid it or seek professional advice.