Web Scraping Legal Guide (UK): GDPR, robots.txt, and the Computer Misuse Act

This is not legal advice. This article provides educational context about how UK law has been applied to web scraping. For specific legal questions about your use case, consult a solicitor qualified in data protection and information technology law.

The three legal frameworks relevant to UK web scraping

Web scraping in the UK is assessed under three overlapping legal frameworks:

1. Computer Misuse Act 1990 — criminalises unauthorised access to computer material
2. UK GDPR and Data Protection Act 2018 — governs processing of personal data
3. Database Directive — protects substantial investment in database creation (implemented by the Copyright and Rights in Databases Regulations 1997)

Most web scraping of publicly accessible, non-personal data sits in a grey zone that these laws have not fully resolved. This article explains where the clear lines are and where the uncertainty remains.

The Computer Misuse Act 1990

The CMA makes it a criminal offence to:

• Access computer material without authorisation (Section 1)
• Access computer material to commit further offences (Section 2)
• Impair the operation of a computer (Section 3)

When does scraping become “unauthorised access”?

The law has not produced definitive case law on this in the UK scraping context. The most relevant principle from existing case law: access is “unauthorised” when the owner has communicated that the particular type of access is not permitted, and the scraper ignores this.

Practical implications:

• Scraping public data with no login requirement: Low CMA risk (the content is publicly accessible)
• Scraping data behind a login you are not authorised to have: High CMA risk
• Scraping in violation of clearly stated ToS that prohibited scraping: Medium-to-high risk
• Causing server load that disrupts normal service: Potential Section 3 risk (impairing computer operation)

The key question is “authorised by whom?” If the website operator has not granted access permission, but the content is publicly accessible, the CMA’s “authorisation” concept becomes contested.

UK GDPR and personal data scraping

This is where the clearest rules apply. Personal data includes any information that can identify a living individual — names, email addresses, phone numbers, photographs, LinkedIn profiles, and often professional titles when combined with employer names.

Under UK GDPR:

Lawful basis is required for processing personal data Scraping and storing personal data requires a lawful basis from Article 6 UKGDPR:

• Consent (rare in B2C scraping — you rarely have consent from individuals)
• Legitimate interests (most commonly argued basis for B2B data scraping)
• Legal obligation or vital interests (rarely applicable to commercial scraping)

Legitimate interests for B2B data: The ICO has indicated that scraping publicly available business contact information (company name, generic business email, professional role) for B2B marketing may fall under legitimate interests, provided:

• You conduct a Legitimate Interests Assessment (LIA)
• The data is truly publicly available (LinkedIn public profiles, company websites)
• You provide a clear opt-out mechanism in every communication
• You do not use scraped data to contact individuals in a way that would be unexpected or intrusive

What is clearly prohibited:

• Scraping personal data from platforms where users have not made it publicly available (private Facebook groups, private LinkedIn connections, closed forums)
• Scraping special category data (health, political views, ethnicity) — requires explicit consent
• Selling scraped personal data lists without a clear lawful basis

ICO enforcement context: The ICO has fined companies for processing scraped data without proper lawful basis. In practice, the ICO tends to focus on organisations processing data at scale commercially. Small-scale research or journalism use of publicly available personal data is less likely to attract enforcement, but is not exempt.

robots.txt: what it is and isn’t

robots.txt is not a legally binding document under UK law. It is a technical convention. Violating robots.txt instructions is not automatically illegal.

However, robots.txt has legal relevance in several ways:

• It is evidence that the website operator communicated their scraping preferences
• Deliberately ignoring robots.txt, combined with other factors (ToS violation, causing load), strengthens a case that access was “unauthorised” under the CMA
• Some terms of service explicitly require compliance with robots.txt, making violation a breach of contract

Practical approach: Follow robots.txt by default. If you have a specific reason to deviate (e.g., scraping your own site, or the robots.txt is clearly overly broad), document your reasoning and ensure compliance with other applicable law.

Terms of Service: contract law implications

Website Terms of Service are contracts. In the UK, you enter into a website’s ToS when you use the site. If those ToS prohibit scraping, scraping in breach of them is a breach of contract.

The available remedies for breach of ToS are civil (not criminal): an injunction to stop scraping, and damages. Websites that want to stop scrapers typically pursue injunctive relief, not criminal prosecution.

Enforceability caveats:

• ToS must be clearly communicated (not buried in a 50-page document nobody reads before using a public site)
• Prohibitions on scraping publicly accessible data face “shrink-wrap” enforceability questions
• Courts have varied in how they treat ToS as meaningful contracts for general site usage

The safest approach: read the ToS of sites you plan to scrape at scale commercially. If scraping is prohibited, either seek a data licence or consult a solicitor before proceeding.

The hiQ v. LinkedIn precedent (US context, relevant to UK businesses)

The hiQ Labs v. LinkedIn case (US 9th Circuit) ruled that scraping publicly accessible data does not violate the US Computer Fraud and Abuse Act because the data is not “protected” — anyone can access it. This is US law and does not directly apply in the UK.

However, it is influential because:

• Many legal scholars cite it in arguing that scraping publicly accessible data should not be treated as “hacking”
• UK companies operating globally face US law when their scrapers target US sites (or when US companies sue UK scrapers in US courts)
• The principle — that publicly accessible data is accessible by anyone, including machines — has influenced UK policy discussions

UK equivalent case law: The UK does not have a direct hiQ equivalent. The 2004 case British Horseracing Board v. William Hill addressed database rights (not computer misuse) and was significant for establishing what constitutes a “substantial part” of a database under the Database Directive.

Copyright and database rights

Published web content (articles, product descriptions, images) is copyright-protected. Scraping and republishing copyrighted content without permission is copyright infringement.

The Database Directive creates a “database right” for databases that represent “substantial investment” in obtaining, verifying, or presenting their contents. Scraping the entirety or a substantial part of a database that qualifies for this protection is infringement.

Practical implications:

• Scraping facts (prices, names, public records) is generally not copyright infringement — facts are not copyrightable
• Scraping verbatim text content and republishing it is copyright infringement
• Scraping a significant portion of a price database at regular intervals to create a competing database may infringe database rights

A practical legal risk matrix

What “defensible scraping” looks like

A commercially defensible scraping operation in the UK:

1. Targets publicly accessible data without circumventing authentication
2. Respects robots.txt Crawl-delay and Disallow directives
3. Has reviewed and does not materially violate the target’s Terms of Service
4. Does not scrape personal data without a documented lawful basis (LIA if using legitimate interests)
5. Has a data retention and deletion policy for any personal data collected
6. Scrapes at a rate that does not impair normal site operation
7. Has a process for responding to cease-and-desist requests

If your use case involves scraping personal data at commercial scale, commissioning a Legitimate Interests Assessment from a data protection solicitor before launch is strongly recommended.