Scraping JavaScript-Rendered Pages: Why Requests Fails and What to Use Instead

The problem: what you get vs what you need

You find a product catalogue at https://example.com/products. You write:

import requests

response = requests.get("https://example.com/products")
print(response.text)

The output is HTML, but it contains something like:

<div id="app"></div>
<script src="/static/js/main.chunk.js"></script>

No product data. Empty containers. The products exist — you can see them in your browser — but they are not in the HTML.

This is a JavaScript-rendered page. The content is loaded by JavaScript after the initial HTML is served. requests retrieves the initial HTML only; it does not execute JavaScript.

What the browser actually does

When Chrome loads https://example.com/products:

1. Makes an HTTP request, receives the initial HTML (containing <div id="app"></div>)
2. Parses the HTML, downloads referenced JavaScript files
3. Executes the JavaScript (React, Vue, or another framework)
4. The JavaScript makes an API call: fetch('/api/products?page=1')
5. The API returns JSON data: [{name: "Widget", price: 29.99}, ...]
6. The JavaScript renders the JSON data into DOM elements inside <div id="app">
7. You see products in the browser

requests stops after step 1. It has no JavaScript engine.

Approach 1: Find and call the API directly

The cleanest solution, when available. The JavaScript making the API call is visible to you in Chrome DevTools.

How to find the API:

1. Open Chrome DevTools → Network tab
2. Filter by “Fetch/XHR” (API calls)
3. Load the page and watch the requests
4. Find the request that returns product data (look for JSON responses)
5. Right-click → Copy → Copy as cURL

If the API returns JSON without authentication, you can call it directly:

import requests

response = requests.get(
    "https://example.com/api/products",
    params={"page": 1, "per_page": 100},
    headers={"Accept": "application/json"}
)
data = response.json()
products = data["products"]

This approach is 10-100x faster than using a headless browser. Always try this first before reaching for Playwright.

Limitations:

• The API may require authentication (cookies from a login session, API keys)
• The API may be undocumented and change without notice
• Some SPAs use GraphQL, which requires a specific query format

Approach 2: Playwright (headless browser)

When the API is authenticated or impractical to call directly, use Playwright to control a real browser:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    
    # Navigate to the page
    page.goto("https://example.com/products")
    
    # Wait for the JavaScript to render the products
    page.wait_for_selector(".product-card", timeout=10000)
    
    # Now extract from the fully rendered DOM
    products = page.query_selector_all(".product-card")
    for product in products:
        name = product.query_selector(".name").inner_text()
        price = product.query_selector(".price").inner_text()
        print(f"{name}: {price}")
    
    browser.close()

The critical step: page.wait_for_selector(".product-card") — this tells Playwright to wait until that CSS selector appears in the DOM, which means JavaScript has finished rendering the products. Without this wait, you might extract the DOM before the products load.

Approach 3: Intercept the API call from within Playwright

You can run Playwright while intercepting the network request, capturing the API response without having to reverse-engineer the API yourself:

from playwright.sync_api import sync_playwright
import json

products = []

def handle_response(response):
    if "/api/products" in response.url:
        data = response.json()
        products.extend(data.get("products", []))

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    
    # Intercept all API responses
    page.on("response", handle_response)
    
    page.goto("https://example.com/products")
    page.wait_for_load_state("networkidle")
    
    browser.close()

print(f"Captured {len(products)} products from API")

This approach captures the clean JSON data directly without HTML parsing. It is faster than DOM extraction and produces cleaner data.

Handling infinite scroll

Many product pages load more content as you scroll:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")
    
    previous_height = 0
    while True:
        # Scroll to the bottom of the page
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        page.wait_for_timeout(2000)  # Wait for new items to load
        
        current_height = page.evaluate("document.body.scrollHeight")
        if current_height == previous_height:
            break  # No new content loaded
        previous_height = current_height
    
    # All items are now loaded
    products = page.query_selector_all(".product-card")
    print(f"Found {len(products)} total products")
    browser.close()

Handling “Load More” buttons

Some sites use a button instead of infinite scroll:

while True:
    # Check if "Load More" button exists
    load_more = page.query_selector("button.load-more")
    if not load_more:
        break  # No more pages
    
    # Click it
    load_more.click()
    page.wait_for_timeout(1500)  # Wait for new items to appear

Performance considerations

Headless browsers are expensive. Each browser instance uses 150-300MB RAM. Scraping 10,000 pages with a single Playwright instance takes approximately 8-12 hours at a polite rate.

For large-scale JavaScript rendering, consider:

1. Async Playwright pools — run 5-10 browser instances in parallel
2. Apify — managed cloud platform that handles JavaScript rendering at scale
3. Bright Data Scraping Browser — proxied browser with stealth built in
4. Zyte SmartProxy — managed rendering that appears as real browser traffic

For under 1,000 pages/day, a single Playwright instance is manageable. For 100,000+ pages/day, managed cloud rendering becomes more economical.

The decision tree

Can you find the API call in Chrome DevTools → Network → XHR?
  YES → Does the API call require authentication you can't replicate?
          NO → Call the API directly with requests. Done.
          YES → Use Playwright and intercept the network response.
  NO  → Is the content rendered client-side at all?
          Check: right-click → View Page Source
          Data in source → use Scrapy/requests
          Data not in source → use Playwright DOM extraction

Quick reference: wait strategies in Playwright

Avoid wait_for_timeout() as your primary wait strategy — it is fragile. Prefer selector-based waits that trigger exactly when the content is ready.