HTTP Status Codes

What HTTP status codes are

Every HTTP request receives a response with a status code indicating what happened. The code is a three-digit number organised into categories:

• 2xx — Success: The request was processed successfully
• 3xx — Redirection: The resource has moved; follow the redirect
• 4xx — Client error: The request has a problem (bad syntax, authentication required, access denied)
• 5xx — Server error: The server failed to process the request

For web scrapers, the specific code returned determines whether to accept the data, retry the request, change strategy, or skip the URL.

Status codes that matter for scraping

200 OK

The request succeeded and the response body contains the requested data. In most cases, this is what you want.

Watch for false 200s: Some sites return HTTP 200 with an empty body, a CAPTCHA page, or a login redirect page instead of blocking with 403/429. Always validate that the response body contains the expected data, not just that the status code is 200.

response = requests.get(url, headers=headers)
if response.status_code == 200:
    # Validate the content, not just the status code
    if "product-name" not in response.text:
        # Likely served a bot detection page
        log.warning(f"200 with suspicious content at {url}")

301 / 302 Moved Permanently / Found

The URL has redirected. requests and most HTTP clients follow redirects automatically. In Scrapy, redirects are handled by the RedirectMiddleware (enabled by default).

Infinite redirect detection: Some anti-bot systems redirect scrapers in a loop. Scrapy’s REDIRECT_MAX_TIMES = 20 (default) prevents infinite redirect loops.

403 Forbidden

The server understood the request but refuses to process it. For scrapers, this typically means:

• Your IP is blocked
• Your User-Agent is blocked
• The request is missing required headers (e.g., missing Referer or Cookie)

First response to 403:

1. Retry with a different User-Agent
2. Check if adding a Referer header (simulating navigation from Google or the site’s homepage) resolves it
3. If persistent: switch to a different IP (proxy rotation)

404 Not Found

The requested URL does not exist. For scrapers following links, this is normal — pages are removed. Log 404s and continue.

Mass 404s on a previously working site: May indicate your IP is being served fake 404s as a soft ban. Check the response body — a real 404 has a normal error page; a soft ban may return a minimal response.

429 Too Many Requests

The server is rate limiting you. This is a direct signal to slow down.

Standard response:

1. Stop requests immediately
2. Check the Retry-After header for how long to wait
3. Wait the specified time (plus buffer) before retrying
4. Reduce your request rate going forward

if response.status_code == 429:
    retry_after = int(response.headers.get("Retry-After", 60))
    logger.warning(f"Rate limited. Waiting {retry_after}s")
    time.sleep(retry_after + random.uniform(5, 15))
    return fetch_with_retry(url)  # retry after waiting

In Scrapy, use AUTOTHROTTLE_ENABLED = True to automatically adjust request rate based on server response times, which reduces 429s preemptively.

503 Service Unavailable

The server is temporarily unable to handle the request. For scrapers, this often means:

• The server is genuinely overloaded (real 503)
• The CDN/anti-bot layer is rejecting the request (disguised block)

A genuine 503 resolves with a simple retry after delay. A block disguised as 503 persists. After 3–4 retries still returning 503, treat it as a block and switch IPs.

407 Proxy Authentication Required

Your proxy requires authentication but the credentials weren’t sent correctly. Check proxy URL format.

522 / 524 (Cloudflare)

These are Cloudflare-specific error codes indicating the origin server timed out. Usually means the target site itself is having issues, not related to your scraper.

Building a status code handler in Python

import requests
import time
import random
from dataclasses import dataclass

@dataclass
class FetchResult:
    url: str
    status: int
    content: str | None
    error: str | None

def fetch_with_handling(url: str, headers: dict, max_retries: int = 3) -> FetchResult:
    for attempt in range(max_retries):
        try:
            response = requests.get(url, headers=headers, timeout=20)
            
            if response.status_code == 200:
                return FetchResult(url, 200, response.text, None)
            
            elif response.status_code == 429:
                wait = int(response.headers.get("Retry-After", 60))
                time.sleep(wait + random.uniform(10, 30))
                continue  # retry
            
            elif response.status_code == 403:
                # IP or UA blocked — don't retry without changing something
                return FetchResult(url, 403, None, "Access forbidden — rotate IP/UA")
            
            elif response.status_code == 404:
                return FetchResult(url, 404, None, "Page not found")
            
            elif response.status_code in (500, 502, 503, 504):
                # Server error — retry with backoff
                time.sleep(2 ** attempt + random.uniform(0, 5))
                continue
            
            else:
                return FetchResult(url, response.status_code, None, 
                                  f"Unexpected status: {response.status_code}")
        
        except requests.Timeout:
            time.sleep(2 ** attempt)
            continue
    
    return FetchResult(url, -1, None, "Max retries exceeded")

Status codes in Scrapy

Scrapy’s default configuration only processes 200 responses (and follows 3xx redirects). Non-200 responses are dropped unless you configure the spider or middleware to handle them.

To handle specific non-200 codes:

class MySpider(scrapy.Spider):
    handle_httpstatus_list = [403, 404, 429]  # process these in parse()
    
    def parse(self, response):
        if response.status == 429:
            # Retry with delay
            yield scrapy.Request(
                response.url, 
                callback=self.parse,
                dont_filter=True,
                meta={"download_delay": 60}
            )