Rate Limiting

What rate limiting is

A rate limiter is server logic that tracks how many requests a client (identified by IP address, session token, or both) has made in a time window, and returns an error when the limit is exceeded.

Common rate limit configurations:

• 60 requests per minute per IP
• 1,000 requests per hour per authenticated user
• 10 requests per second globally (for high-traffic APIs)

When a scraper exceeds the rate limit, it receives:

• HTTP 429 Too Many Requests — the standard response code for rate limiting
• Sometimes HTTP 503 Service Unavailable — used by some anti-bot systems
• Sometimes HTTP 403 Forbidden — after repeated violations

The Retry-After header

A well-implemented rate limiter includes a Retry-After header in 429 responses:

HTTP/1.1 429 Too Many Requests
Retry-After: 120
Content-Type: application/json

{"error": "Rate limit exceeded. Please wait 120 seconds before retrying."}

The value can be:

• An integer (seconds to wait): Retry-After: 120
• An HTTP date (timestamp when you can retry): Retry-After: Fri, 31 May 2026 10:00:00 GMT

A well-designed scraper reads this header and waits the specified time rather than retrying immediately.

Scrapy’s AUTOTHROTTLE extension

Scrapy’s built-in AUTOTHROTTLE dynamically adjusts request rate based on server response times. Configuration:

# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0    # Initial download delay in seconds
AUTOTHROTTLE_MAX_DELAY = 60.0     # Maximum download delay
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0  # Average concurrent requests per server
AUTOTHROTTLE_DEBUG = False        # True to see delay adjustments in logs

AUTOTHROTTLE targets a concurrency level (default 1.0 concurrent requests per server) and adjusts delays based on whether the server is responding faster or slower than expected. When the server slows down (higher latency, 503 responses), AUTOTHROTTLE increases the delay. When it recovers, AUTOTHROTTLE speeds back up.

This is more sophisticated than a fixed delay because it responds to actual server health.

robots.txt Crawl-delay

robots.txt can specify a requested delay between requests:

User-agent: *
Crawl-delay: 10

This is a request, not a technical enforcement. Scrapy respects Crawl-delay by default when ROBOTSTXT_OBEY = True. Other libraries do not automatically read this — you must implement it manually.

Ignoring Crawl-delay when scraping aggressively is ethically poor practice and increases blocking risk.

Handling rate limiting in code

Python requests with exponential backoff:

import requests
import time

def get_with_retry(url, headers, max_retries=5):
    for attempt in range(max_retries):
        response = requests.get(url, headers=headers)
        
        if response.status_code == 200:
            return response
        
        if response.status_code == 429:
            retry_after = int(response.headers.get("Retry-After", 2 ** attempt * 10))
            print(f"Rate limited. Waiting {retry_after}s")
            time.sleep(retry_after)
            continue
        
        if response.status_code in [500, 502, 503, 504]:
            time.sleep(2 ** attempt * 5)  # Exponential backoff for server errors
            continue
        
        # Unrecoverable error
        response.raise_for_status()
    
    raise Exception(f"Max retries exceeded for {url}")

Scrapy middleware for 429 handling:

from scrapy.exceptions import NotConfigured
import time

class RateLimitMiddleware:
    def process_response(self, request, response, spider):
        if response.status == 429:
            retry_after = int(response.headers.get('Retry-After', 60))
            spider.logger.info(f"Rate limited. Sleeping {retry_after}s")
            time.sleep(retry_after)
            return request  # Retry the request
        return response

Rate limiting vs IP blocking

Rate limiting and IP blocking are different mechanisms with different implications:

Aspect	Rate Limiting	IP Blocking
HTTP status	429 Too Many Requests	403 Forbidden or no response
Temporary?	Usually (waits are finite)	Often permanent or long-term
Proxy solution	Not usually — the rate is tracked per new IP too	Yes — new IPs are clean
Response	Wait and retry	Use a different proxy
Detection trigger	Request volume	Fingerprinting or repeated violations

If you receive 429 responses: wait and retry. Do not rotate proxies unnecessarily — rate limits often track per API key or user, not per IP.

If you receive 403 responses after previously working requests: the IP may be blocked. Rotating to a fresh proxy resolves this.