Affiliate disclosure: We earn a commission if you sign up through our links. This does not influence our test results or editorial scores. Full disclosure →

How to Scrape a Website Without Getting Blocked: Rate Limiting, Headers, and Proxy Rotation

Last reviewed: 2026-05-01 · 14 min read · WebScrapingTool.net

Why scrapers get blocked

Websites block scrapers to protect their infrastructure and their data. The detection mechanisms they use fall into three categories:

Volume-based detection: Too many requests from one IP in a short time. The server’s rate limiter logs IP X making 500 requests in 2 minutes and blocks it.

Fingerprint-based detection: Your HTTP requests look different from a human browser. Missing or inconsistent headers, HTTP/1.1 where browsers use HTTP/2, no TLS session tickets.

Behaviour-based detection: Mouse movements, JavaScript execution timing, session patterns. Sophisticated anti-bot systems (Cloudflare, PerimeterX, Akamai) track these at the browser level.

This guide covers the first two categories — the most common cases in practice.

1. User-Agent rotation

The User-Agent header identifies your client. A real Chrome browser sends something like:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

Python’s requests library sends python-requests/2.31.0 by default. This is immediately identifiable as a bot.

Simple fix — set a realistic User-Agent:

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
}

response = requests.get("https://example.com/products", headers=headers)

Better — rotate User-Agents:

import requests
import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
]

def get_random_headers():
    return {"User-Agent": random.choice(USER_AGENTS)}

response = requests.get("https://example.com/products", headers=get_random_headers())

Maintain a fresh list of user agents — browser versions update regularly, and using old User-Agent strings is itself a detection signal.

2. Adding realistic request headers

A real browser does not just send a User-Agent. It sends a bundle of headers that together form a recognisable fingerprint:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Cache-Control": "max-age=0",
}

response = requests.get("https://example.com/products", headers=headers)

The Sec-Fetch-* headers were introduced in Chrome 76 and are now sent by all modern browsers. Their absence is a bot signal.

3. Request rate limiting

The simplest way to avoid blocking: don’t request too fast. A human browsing a product catalogue might load 1-2 pages per minute. A scraper loading 300 pages per minute looks very different.

Simple delay:

import time
import random

for url in urls_to_scrape:
    response = requests.get(url, headers=get_random_headers())
    process(response)
    # Random delay between 1 and 4 seconds
    time.sleep(random.uniform(1, 4))

Scrapy AUTOTHROTTLE:

# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

AUTOTHROTTLE adjusts request rate based on server response times — faster when the server is responding quickly, slower when it’s showing signs of load. This is more sophisticated than a fixed delay and reduces blocking risk.

4. Proxy rotation

Even with perfect headers and polite delays, scraping thousands of pages from one IP address will eventually trigger threshold-based blocking. IP rotation is the solution.

Types of proxies:

Datacenter proxies: Cheap (£0.001-0.01/request), fast, but easily detected (IP ranges are known to be datacenter-owned)
Residential proxies: Expensive (£0.05-0.20/request), slower, but appear as real consumer ISP connections
ISP proxies: Hybrid — legitimate ISP-owned IPs, faster than residential, more expensive than datacenter

For most scraping, datacenter proxies are sufficient. For heavily anti-bot-protected sites (e-commerce, social media), residential proxies are needed.

Basic proxy rotation with requests:

import requests
import random

PROXIES = [
    {"http": "http://proxy1:port", "https": "http://proxy1:port"},
    {"http": "http://proxy2:port", "https": "http://proxy2:port"},
    {"http": "http://proxy3:port", "https": "http://proxy3:port"},
]

def get_proxy():
    return random.choice(PROXIES)

response = requests.get(
    "https://example.com/products",
    headers=get_random_headers(),
    proxies=get_proxy()
)

Commercial proxy services with Scrapy: Most proxy providers offer Scrapy-compatible middleware. Bright Data example:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 1,
}

# Use Bright Data's rotating endpoint
PROXY = "http://brd-customer-{customer_id}-zone-{zone}:{password}@brd.superproxy.io:22225"

5. Session management and cookies

Many sites set cookies on first visit that are expected on subsequent requests. Sending requests without the expected cookies is a bot signal.

import requests

session = requests.Session()

# First visit - get cookies
session.get("https://example.com/", headers=get_random_headers())

# Subsequent requests carry the cookies automatically
response = session.get("https://example.com/products", headers=get_random_headers())

Using requests.Session() preserves cookies across requests, matching real browser behaviour.

6. Handling 429 (Too Many Requests) and 503 responses

A well-designed scraper detects blocks and backs off:

import requests
import time

def scrape_with_backoff(url, max_retries=5):
    for attempt in range(max_retries):
        response = requests.get(url, headers=get_random_headers())
        
        if response.status_code == 200:
            return response
        elif response.status_code == 429:
            # Rate limited — wait and retry
            wait_time = 2 ** attempt * 10  # 10s, 20s, 40s, 80s, 160s
            print(f"Rate limited. Waiting {wait_time}s before retry {attempt+1}")
            time.sleep(wait_time)
        elif response.status_code == 503:
            # Service unavailable — wait and retry
            time.sleep(30)
        else:
            print(f"Unexpected status {response.status_code} for {url}")
            return None
    
    print(f"Failed after {max_retries} retries: {url}")
    return None

7. What you cannot bypass without a headless browser

The techniques above handle the majority of blocking cases. They do not help with:

JavaScript challenges: Cloudflare’s JS challenge presents JavaScript that must execute before the real page loads. Only a browser can pass this.
Browser fingerprinting: If the anti-bot system checks navigator.webdriver, canvas fingerprints, or WebGL data, these require browser-level spoofing.
CAPTCHA: No proxy or header trick bypasses a CAPTCHA. Use a headless browser with a CAPTCHA solver service.

For these cases, Playwright with playwright-stealth is the starting point. For heavily protected enterprise targets, commercial solutions (Apify’s SmartProxy, Bright Data’s Scraping Browser) may be the practical choice.