Web scraping fundamentals user-agent HTTP headers bot detection web scraping anti-bot

User-Agent

What the User-Agent header is

The User-Agent (UA) is an HTTP header sent with every web request. It tells the server what client is making the request. A real Chrome browser on Windows 10 sends something like:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

This string encodes the browser engine (Gecko, WebKit), operating system, and browser version. Servers use it for analytics, compatibility decisions, and bot detection.

Default User-Agents from common scraping libraries

Library / Tool	Default User-Agent
Python requests 2.31	`python-requests/2.31.0`
Scrapy 2.11	`Scrapy/2.11.0 (+https://scrapy.org)`
curl	`curl/8.4.0`
wget	`Wget/1.21.4`
httpx (Python)	`python-httpx/0.27.0`
Chrome 124 (real browser)	`Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...`

python-requests/2.31.0 is immediately identifiable as a scraping library. Any site with basic bot detection will flag this. Setting a realistic User-Agent is the first, most basic evasion step.

How to set a User-Agent

requests:

import requests
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}
response = requests.get("https://example.com", headers=headers)

Scrapy (settings.py):

USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

Playwright:

browser = p.chromium.launch(headless=True)
page = browser.new_page(user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)...")

Rotating User-Agents

Using a single UA, even a realistic one, becomes a signal when the same UA string makes thousands of requests. Rotating across a list of realistic UAs reduces this pattern:

import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
]

def get_headers():
    return {"User-Agent": random.choice(USER_AGENTS)}

Keep UA lists updated — using UA strings for browser versions from 2020 is itself detectable. Refresh the list periodically to reflect current browser releases.

User-Agent and robots.txt

robots.txt can specify directives per User-Agent:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /private/
Crawl-delay: 5

In this example, Googlebot has no restrictions. All other agents are blocked from /private/ and asked to wait 5 seconds between requests. If you set your UA to “Googlebot,” you inherit Googlebot’s rules — but this is both ethically questionable and detectable (Google owns Googlebot and its IP range is known).

What a UA string does not tell you (and sophisticated detection does)

A realistic User-Agent string is table stakes, not a silver bullet. Sophisticated anti-bot systems also check:

TLS fingerprint (how the SSL handshake is performed)
HTTP/2 settings frames (browsers send specific settings; libraries don’t match exactly)
Header order (browsers send headers in a consistent order; libraries may vary)
JavaScript execution (only relevant if the site serves a JS challenge)
Behavioural signals (mouse movement, scroll patterns, click timing)

Setting a realistic UA eliminates the most basic detection layer. Defeating advanced detection requires tools like curl-impersonate, playwright-stealth, or commercial solutions that handle full browser fingerprinting.

Related concepts

→ robots.txt → rate limiting → anti bot detection → rotating proxy