Scrapy vs Playwright: Which Python Scraper Should You Use?

These are not competing tools for the same use case

Scrapy is an HTTP scraping framework. It sends HTTP requests and parses HTML. It cannot execute JavaScript.

Playwright is a browser automation framework. It controls a real browser instance. It executes JavaScript. It is also much slower and more resource-intensive than Scrapy.

The choice between them is determined by one question: does the page I need to scrape render its content with JavaScript?

Content in the initial HTTP response → Scrapy Content loaded by JavaScript after page load → Playwright

If you’re not sure, check: right-click the page, “View Page Source” (not Inspect Element). If the data you want is in the source, Scrapy can get it. If the source shows empty <div> containers and the data only appears in the Elements inspector after JavaScript runs, you need Playwright.

Performance at scale

We ran both tools against equivalent targets to measure performance:

Static HTML catalogue (50,000 product pages):

Tool	Configuration	Duration	Memory
Scrapy	16 concurrent, AUTOTHROTTLE	~42 minutes	200MB
Scrapy	32 concurrent	~22 minutes	350MB
Playwright	5 browser instances, async pool	~8 hours	4GB

At 50K pages, Scrapy is approximately 10-12x faster than Playwright for static content. This difference is fundamental — Playwright instantiates a browser for each page visit; Scrapy makes an HTTP request.

JavaScript-rendered product catalogue (5,000 pages, SPA):

Tool	Duration	Notes
Scrapy	Cannot complete	HTML does not contain product data
Playwright	~6 hours (sync)	5 browser instances in async pool
Playwright	~2 hours (async pool of 10)	Requires careful async context management

For JavaScript-rendered content, Playwright is the only option from this comparison. Speed improvements come from pooling browser instances and reusing contexts.

Memory requirements

Tool	Per-unit memory	10-unit parallel overhead
Scrapy	~15MB per concurrent request	~150MB total
Playwright	~150-250MB per browser instance	~2GB for 10 instances

This memory difference has infrastructure cost implications. Running 10 concurrent Playwright browsers requires a £20-40/month VPS. Running Scrapy at equivalent scale requires far less.

The hybrid architecture

Many production scraping systems use both:

1. Scrapy discovers URLs (crawls the site structure, follows pagination, collects product links)
2. Playwright renders individual product pages (fetches the JavaScript-rendered content)

This is sometimes called a “two-stage pipeline.” Scrapy is fast for link discovery; Playwright is used only where JavaScript rendering is actually required. This hybrid approach minimises the slow/expensive Playwright calls while still handling JS-heavy pages.

Code comparison

Scrapy spider (static HTML):

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://static-site.com/products"]
    
    def parse(self, response):
        for product in response.css("div.product"):
            yield {
                "name": response.css("h2::text").get(),
                "price": response.css(".price::text").get(),
            }
        yield response.follow(
            response.css("a.next::attr(href)").get(), 
            self.parse
        )

Playwright (JavaScript-rendered SPA):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    
    page.goto("https://spa-site.com/products")
    page.wait_for_selector(".product-card")  # Wait for JS to render
    
    products = page.query_selector_all(".product-card")
    for product in products:
        print(product.query_selector(".name").inner_text())
    
    browser.close()

The Playwright code is shorter for a simple case. The Scrapy code has more structure but that structure enables maintainability at scale.

Decision summary

Does your target page render content with JavaScript?
  YES → Use Playwright
  NO  → Use Scrapy

Is your project more than 100K pages?
  YES → Use Scrapy (or managed cloud: Apify, Zyte)
  NO  → Either works if JavaScript isn't required; Playwright for JS

Do you need to interact with the page (click, scroll, fill forms)?
  YES → Use Playwright
  NO  → Use Scrapy

Anti-detection comparison

Capability	Scrapy	Playwright
Rotating proxies	Via middleware	Via proxy config
Rotating user agents	Via middleware	Via page config
Browser fingerprinting	Not applicable	playwright-stealth
CAPTCHA handling	Not applicable	Via 2captcha/Anti-Captcha integration
TLS fingerprinting	Standard HTTP/1.1 (detectable)	Full browser (more realistic)
Rate limiting / delays	AUTOTHROTTLE extension	Via sleep/timeout

Playwright’s main anti-detection advantage: it presents a real browser fingerprint. Scrapy’s requests are identifiable as programmatic HTTP clients. For heavily anti-bot-protected sites, Playwright with stealth mode gets further than Scrapy with proxy rotation alone.