Headless Browser
What a headless browser is
A headless browser is a complete browser engine (rendering, JavaScript execution, network stack) running without a visible window. It can:
- Navigate to URLs
- Execute JavaScript (including React, Vue, Angular)
- Wait for AJAX requests to complete
- Click elements, fill forms, scroll pages
- Take screenshots and generate PDFs
- Manage cookies and sessions
From the server’s perspective, a correctly configured headless browser looks identical to a regular browser visit.
Primary tools
Playwright (Microsoft, 2020):
- Controls Chromium, Firefox, and WebKit
- Python, JavaScript, TypeScript, Java, .NET APIs
- Better async support and more features than Puppeteer
- Actively developed as of 2026
Puppeteer (Google, 2017):
- Controls Chromium primarily
- JavaScript/TypeScript only
- Older and more mature ecosystem but slower development pace
- Still widely used in Node.js scraping stacks
Selenium (Older standard):
- Controls Chrome, Firefox, Edge, Safari
- Java, Python, JavaScript, Ruby, C#
- More complex API, primarily used for browser testing
- Less commonly used for scraping in 2026
Memory requirements
Headless browsers are significantly more resource-intensive than HTTP-only scrapers:
| Tool | Memory per instance | Suitable for |
|---|---|---|
| Python requests | ~50MB process | Millions of static pages |
| Scrapy | ~150MB per worker | Millions of static pages |
| Playwright (single browser) | ~200-400MB | Up to ~50K JS pages/day |
| Playwright (5 browser pool) | ~1-2GB | Up to ~250K JS pages/day |
At scale, memory constraints are the primary limitation for headless browser scraping.
Detection and evasion
Headless browsers can be detected by:
navigator.webdriver JavaScript property:
Headless Chrome sets navigator.webdriver = true. Real browsers have false. Many anti-bot scripts check this first.
Fix: playwright-stealth library patches this and 20+ other automation signals.
Canvas and WebGL fingerprinting: Headless browsers render canvas elements slightly differently from headed browsers. Anti-bot systems collect canvas fingerprints and compare against known patterns.
Font enumeration: Browsers expose installed fonts through JavaScript. Headless environments often have different font sets than real user machines.
TLS fingerprinting: The TLS handshake details (cipher suites, extensions, order) differ between Playwright and a real Chrome browser.
Installation
Playwright (Python):
pip install playwright
playwright install chromium # Downloads Chromium (~150MB)
Playwright (Node.js):
npm install playwright
npx playwright install chromium
The downloaded browser is managed by Playwright separately from any system-installed Chrome. This ensures version consistency across environments.
Cloud and serverless headless browsers
Running headless browsers in the cloud requires specific configuration:
AWS Lambda: Playwright with --no-sandbox flag; custom Chromium layers are available on GitHub (chromium-layer).
Docker: Use the official Playwright Docker image (mcr.microsoft.com/playwright/python:v1.43.0-jammy) which includes all system dependencies.
Managed services:
- Apify uses Playwright internally and manages the browser infrastructure
- Bright Data Scraping Browser is a cloud-hosted proxy+browser combination
- ScrapingBee provides a headless browser API
For teams that want headless browser capability without managing browser infrastructure, managed services are increasingly competitive with self-hosted solutions.