Proxy Rotation

What proxy rotation is

Every HTTP request originates from an IP address. Anti-bot systems use IP reputation and request frequency per IP to identify and block scrapers. A single IP making 10,000 requests to the same site in an hour is a clear scraping signal.

Proxy rotation routes each request (or each session) through a different IP address. From the target site’s perspective, the requests look like they come from many different users rather than one automated source.

Types of proxies:

Type	Source	Detection risk	Cost
Datacenter	Cloud providers (AWS, GCP, DO)	High — IP ranges are known	Low ($1–3/GB)
Residential	Consumer ISP addresses	Low — look like real users	Medium ($8–15/GB)
Mobile	Mobile carrier IPs	Very low — high trust signals	High ($15–30/GB)
ISP proxies	ISP-allocated IPs with datacenter hosting	Low	Medium ($5–10/GB)

How proxy rotation works technically

Per-request rotation: Each HTTP request uses a different proxy from your pool. Simple to implement, but creates inconsistent sessions — the same user appears to change location with every request, which can trigger session-based detection.

Per-session rotation: A proxy is assigned for a complete user session (login → browse → extract → logout). Session consistency looks more like real user behaviour. Required for sites with login-based access.

Sticky sessions: Many proxy providers offer sticky sessions — a single IP is maintained for a configurable duration (5–30 minutes). This balances rotation (eventually changing IPs) with session consistency.

Implementation patterns

Manual pool with requests:

import requests
import random
import time

proxies = [
    "http://user:pass@proxy1.example.com:8080",
    "http://user:pass@proxy2.example.com:8080",
    "http://user:pass@proxy3.example.com:8080",
]

def get_with_rotation(url, max_retries=3):
    for attempt in range(max_retries):
        proxy = random.choice(proxies)
        try:
            response = requests.get(
                url,
                proxies={"http": proxy, "https": proxy},
                timeout=15
            )
            if response.status_code == 200:
                return response
        except requests.exceptions.ProxyError:
            # Proxy is down — try another
            continue
        
        time.sleep(2 ** attempt)  # exponential backoff
    
    raise Exception(f"All proxy attempts failed for {url}")

Rotating proxy service (single endpoint): Services like Bright Data, Smartproxy, and Oxylabs provide a single proxy endpoint that automatically rotates IPs on each request. You configure one proxy URL and get automatic rotation:

# Bright Data rotating residential proxy — single endpoint
proxy = {
    "http": "http://user:pass@brd.superproxy.io:22225",
    "https": "http://user:pass@brd.superproxy.io:22225"
}

response = requests.get("https://target.com/data", proxies=proxy)
# Each call automatically uses a different residential IP

Scrapy proxy middleware:

# settings.py
ROTATING_PROXY_LIST = [
    "http://user:pass@proxy1.example.com:8080",
    "http://user:pass@proxy2.example.com:8080",
    # ...
]

DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}

The scrapy-rotating-proxies package handles proxy selection and ban detection automatically.

Residential vs. datacenter proxies

Datacenter proxies are IPs from AWS, GCP, Azure, DigitalOcean, and other cloud providers. They are cheap and fast but easy to detect:

• IP ranges are published (AWS publishes its IP ranges at ip-ranges.amazonaws.com)
• Multiple scraping requests from the same datacenter IP range is a known pattern
• Effective for sites with basic bot protection; blocked immediately by Cloudflare, Akamai, and Imperva

Residential proxies are IPs belonging to real consumer devices — ISP subscribers who have installed software (often through VPNs, browser extensions, or apps) that routes other traffic through their connection. They:

• Look like real user traffic to target sites
• Are not listed in cloud IP ranges
• Cost significantly more ($8–15/GB vs $1–3/GB)
• Have variable quality (bandwidth, latency, availability)

For e-commerce, social media, and financial sites with serious bot protection: residential proxies are necessary. For sites with light or no anti-bot protection: datacenter proxies are sufficient and much cheaper.

When proxy rotation isn’t enough

Proxy rotation prevents IP-based blocking. Sites with advanced bot detection also fingerprint:

• Browser TLS settings
• HTTP/2 header order
• Canvas and WebGL rendering
• Font enumeration
• navigator.webdriver status

Rotating proxies does not solve fingerprinting. For high-protection targets, you need both proxy rotation and browser fingerprint evasion (Playwright-stealth, or a managed scraping browser like Bright Data’s Scraping Browser or Apify’s Puppeteer-stealth).

Cost at scale

At 1 million residential proxy requests (assuming average 200KB page response = 200GB):

• Bright Data residential: 200GB × $8.40/GB = $1,680
• Smartproxy: 200GB × $7/GB = $1,400
• Apify (residential proxies included in plan): $499/month Scale plan

For very high volume (10M+ requests/month), bulk pricing from providers reduces cost to $5–6/GB. At that volume, evaluate whether a managed scraping platform (Apify, Zyte) or a custom infrastructure with direct proxy contracts is more cost-effective.