JSON API Scraping

What JSON API scraping is

Most modern websites do not render data server-side into HTML. Instead, they load a skeleton HTML page and then fetch data via JavaScript using XHR or fetch calls to JSON API endpoints. The browser renders this data into the DOM after the page loads.

When you scrape HTML with Playwright or Selenium, you’re scraping the rendered result. But if you can identify the underlying API endpoint that the browser called to get that data, you can call the same endpoint directly — getting clean JSON without any browser automation overhead.

The performance difference:

• HTML scraping with Playwright: ~20–50 pages/minute (browser startup + render time)
• Direct JSON API calls with requests: ~1,000–5,000 pages/minute (pure HTTP)

JSON API scraping is 20–100× faster than browser scraping when applicable.

How to find hidden API endpoints

Step 1: Open DevTools Network tab

In Chrome: F12 → Network tab → filter by “XHR” or “Fetch”

Load the page and watch for requests that return JSON. They typically appear as requests with application/json content type.

Step 2: Identify the data request

Look for requests that return the data you can see on the page. Sort by response size — data endpoints typically return larger responses than tracking/analytics calls.

Right-click the request → “Copy as cURL” to get the exact request headers the browser sent.

Step 3: Replay in Python

import requests

# Replicate exactly what the browser sent
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-GB,en;q=0.9",
    "Referer": "https://example.com/products",
    "X-Requested-With": "XMLHttpRequest",
}

response = requests.get(
    "https://api.example.com/v2/products?category=electronics&page=1",
    headers=headers
)

data = response.json()
products = data["items"]  # structure varies by site

Step 4: Find pagination

JSON APIs almost always paginate. The first response typically contains pagination metadata:

{
  "items": [...],
  "total": 4821,
  "page": 1,
  "per_page": 50,
  "next_page": "/api/products?page=2"
}

Use this to build your pagination loop.

Common JSON API patterns

Cursor-based pagination:

next_cursor = None
all_items = []

while True:
    params = {"cursor": next_cursor} if next_cursor else {}
    response = requests.get("https://api.example.com/items", 
                           headers=headers, params=params)
    data = response.json()
    all_items.extend(data["items"])
    
    next_cursor = data.get("next_cursor")
    if not next_cursor:
        break

GraphQL endpoints: Some sites use GraphQL. You’ll see POST requests to a /graphql endpoint with JSON bodies containing query and variables. These can be replicated the same way — copy the request body from DevTools and POST it directly.

query = """
query ProductSearch($category: String!, $page: Int!) {
  products(category: $category, page: $page) {
    id
    name
    price
    inStock
  }
}
"""

response = requests.post(
    "https://example.com/graphql",
    json={"query": query, "variables": {"category": "electronics", "page": 1}},
    headers={"Content-Type": "application/json", **headers}
)

When JSON API scraping doesn’t work

1. Auth tokens that expire: Some APIs include time-limited tokens in requests (often generated server-side when the page loads). If your API call returns 401 or empty results, check whether the request includes a token that was generated from the HTML page — you may need to fetch the page first, extract the token, then call the API.

2. Browser fingerprinting on the API: Less common, but some APIs check TLS fingerprints or browser-specific headers. If replaying the exact cURL command from DevTools fails, try using curl-cffi (which mimics browser TLS fingerprints) instead of requests.

3. CORS restrictions (don’t affect you): CORS (Cross-Origin Resource Sharing) restrictions apply to browsers, not to server-side HTTP clients. If the browser can call an API endpoint, your Python script can too — CORS is not a barrier to server-side scraping.

4. Encrypted or obfuscated payloads: Rare, but some sites encrypt API request parameters. This requires JavaScript reverse engineering to replicate, which is significantly more complex.

Ethics and rate limiting

JSON API scraping hits backend infrastructure directly — more efficiently than browser scraping, which means you can generate more load. Apply rate limiting (minimum 1 second between requests; adjust based on Retry-After headers) and check robots.txt for API path directives. Direct API access doesn’t change your legal obligations under the site’s terms of service or the Computer Misuse Act.