Selenium

What Selenium is

Selenium is an open-source browser automation framework originally designed for web application testing. Released in 2004, it became the dominant tool for automating browsers before Playwright existed. Selenium WebDriver controls a browser (Chrome, Firefox, Edge, Safari) programmatically via the WebDriver protocol.

For web scraping, Selenium is used when a target page requires JavaScript execution — cases where requests alone retrieves only the static HTML shell, not the dynamically rendered content.

Selenium in Python:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)
driver.get("https://example.com/products")

# Wait for specific element to appear (not just page load)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "product-card"))
)

products = driver.find_elements(By.CLASS_NAME, "product-card")
for product in products:
    name = product.find_element(By.CLASS_NAME, "product-name").text
    price = product.find_element(By.CLASS_NAME, "product-price").text
    print(f"{name}: {price}")

driver.quit()

Selenium vs. Playwright: the 2025 comparison

Playwright has largely superseded Selenium for new scraping projects. The reasons:

Aspect	Selenium	Playwright
API design	Older, verbose	Modern async/await
Speed	Slower (WebDriver protocol overhead)	Faster (CDP direct connection)
Auto-waiting	Explicit waits required	Built-in smart auto-waiting
Browser support	Chrome, Firefox, Edge, Safari	Chrome, Firefox, WebKit (Safari-like)
Language support	Python, Java, C#, JS, Ruby, PHP	Python, JavaScript, Java, C#
Installation	ChromeDriver version must match Chrome	playwright install handles everything
Stealth	No built-in stealth; requires undetected-chromedriver	playwright-stealth or stealth-config
Network interception	Limited	Full request/response interception

When to still use Selenium:

• You have an existing Selenium codebase and migration cost is high
• You need Java, C#, or PHP language support (Playwright supports these but Python/JS are better documented)
• You’re running a testing framework that integrates with Selenium (pytest-selenium, Selenide)
• Your target site detects Playwright’s CDP fingerprint specifically (rare, but documented for some targets)

For new Python scraping projects: Use Playwright. The async API is cleaner, the auto-waiting is more reliable, and network interception enables patterns that are impossible in Selenium.

ChromeDriver versioning: the classic Selenium pain

Selenium requires a ChromeDriver binary whose version matches your installed Chrome version exactly. Chrome auto-updates; ChromeDriver does not. This mismatch is the most common source of Selenium failures:

SessionNotCreatedException: Message: session not created: 
This version of ChromeDriver only supports Chrome version 114
Current browser version is 124.0.6367.201

Fix: Use webdriver-manager to auto-download the matching ChromeDriver:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

Or use Selenium 4.6+ which includes Selenium Manager (automatic driver management built in).

Selenium in Scrapy: the scrapy-selenium pattern

Some scraping projects use Scrapy’s pipeline and scheduling infrastructure with Selenium for rendering. The scrapy-selenium middleware renders pages via Selenium before passing the response to Scrapy’s normal parsing pipeline.

This works but is architecturally awkward — Scrapy is designed for asynchronous HTTP, and Selenium is synchronous and heavy. The result is slow and memory-intensive.

A cleaner alternative for JS-heavy targets: Scrapy-Playwright (scrapy-playwright on PyPI), which integrates Playwright’s async API natively into Scrapy’s async event loop.

Undetected ChromeDriver

Standard Selenium is easily detected by modern anti-bot systems via the navigator.webdriver JavaScript property, which is set to true in automated browsers. undetected-chromedriver is a patched version that overrides this and several other automation signals:

import undetected_chromedriver as uc

driver = uc.Chrome(headless=True)
driver.get("https://protected-site.com")

Success rates against Cloudflare Bot Management and similar systems vary. Playwright + playwright-stealth is generally more effective at evasion in 2025, but undetected-chromedriver remains useful for sites specifically tuned to detect Playwright’s CDP fingerprint.