Robots.txt
What robots.txt is
robots.txt is a plain text file placed at https://example.com/robots.txt. When a web crawler (Googlebot, a Scrapy spider, a headless browser) visits a site, it is expected to first check this file and follow its instructions.
A minimal robots.txt:
User-agent: *
Disallow: /private/
Allow: /public/
Crawl-delay: 10
User-agent: * means “applies to all crawlers.” You can specify individual user agents by name (User-agent: Googlebot). Disallow: lists paths the crawler should not visit. Crawl-delay: is a rate limit suggestion in seconds between requests.
The legal status of robots.txt
robots.txt is technically not a legal document. It is a convention — the Robots Exclusion Standard — that crawlers are expected to follow by convention, not by law.
Key legal context:
In the US: The hiQ Labs v. LinkedIn case (9th Circuit, 2022) ruled that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act (CFAA) merely because the site’s robots.txt disallows it. However, this ruling is specific to publicly accessible, non-authenticated data. Circumventing authentication to access data behind a login is still CFAA-risky even if robots.txt does not explicitly prohibit it.
In the UK: The Computer Misuse Act 1990 prohibits “unauthorised access to computer material.” Scraping in violation of robots.txt alone does not clearly constitute unauthorised access — but violating explicit terms of service, especially to access authenticated content, raises the risk substantially.
Practical implication: Ignoring robots.txt is not automatically illegal. Ignoring robots.txt and violating terms of service and accessing authenticated content is increasingly risky.
robots.txt and Google Search Console
One important non-scraping use of robots.txt: it controls how Googlebot indexes your site. A misconfigured robots.txt (e.g., Disallow: /) will de-index your entire site from Google. Always test changes to robots.txt using the Google Search Console robots.txt tester before deploying.
How to read a robots.txt file programmatically
In Python with Scrapy, robots.txt is respected by default:
# In settings.py
ROBOTSTXT_OBEY = True # default True
If you have a legitimate reason to disable robots.txt obedience (e.g., scraping your own site during testing):
ROBOTSTXT_OBEY = False
With Python’s urllib:
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
can_fetch = rp.can_fetch("*", "https://example.com/private/page")
Common robots.txt patterns in the wild
| Pattern | Meaning | Scraper implication |
|---|---|---|
Disallow: / | Block all paths | High-value data often here; legal risk of scraping increases |
Disallow: /search | Block search results pages | Often to prevent scraping of aggregated data |
Disallow: /api/ | Block API endpoints | May indicate data is available via official API instead |
Crawl-delay: 60 | Wait 60 seconds between requests | If ignored, risk of IP block is high |
| No robots.txt | No crawl directives | Not an invitation to crawl aggressively — terms of service still apply |
The ethical layer
Even where robots.txt is not legally binding, following it is an ethical norm in the scraping community. Ignoring it when scraping a small site can cause load that disrupts the site’s operation — which crosses into territory more clearly covered by law in the UK (Computer Misuse Act) and US (CFAA).
Scraping frameworks (Scrapy, Playwright) all support robots.txt checking. Use it unless you have a specific legal basis for not doing so.