XPath vs CSS Selectors

What they are

CSS selectors use CSS’s element targeting syntax to select DOM elements. They are the same selectors used to style web pages.

XPath (XML Path Language) is a query language for navigating XML/HTML documents. It expresses paths through the document tree.

Both are supported by Scrapy, Beautiful Soup, Playwright, and most other scraping tools.

CSS selector examples

# Select by tag name
response.css("h1")

# Select by class
response.css(".product-title")

# Select by id
response.css("#main-price")

# Select by attribute
response.css("a[href^='/products']")  # href starts with /products
response.css("input[type='text']")

# Select text content
response.css("h1::text").get()

# Select attribute value
response.css("a::attr(href)").get()

# Nested selectors
response.css("div.product-card h2.name::text").get()

# Multiple selectors (OR)
response.css("h1, h2, h3")

# Child combinator
response.css("div.container > p")

XPath examples

# Select by tag name
response.xpath("//h1")

# Select by class (note: XPath uses contains for class matching)
response.xpath("//div[contains(@class, 'product-card')]")

# Select by id
response.xpath("//*[@id='main-price']")

# Select by attribute
response.xpath("//a[starts-with(@href, '/products')]")

# Text selection
response.xpath("//h1/text()").get()

# Attribute value
response.xpath("//a/@href").get()

# Parent traversal (XPath only — CSS cannot go to parent)
response.xpath("//span[@class='price']/parent::div")

# Sibling traversal
response.xpath("//h2/following-sibling::p[1]")

# Text content matching
response.xpath("//button[text()='Add to Cart']")
response.xpath("//p[contains(text(), 'shipping')]")

When to use CSS selectors

Use CSS selectors when:

• The HTML structure is standard (tags, classes, IDs are well-defined)
• You are selecting by class or attribute — CSS syntax is cleaner
• You are comfortable with CSS from frontend development
• You are using Beautiful Soup (soup.select()) or Scrapy (response.css())

CSS selectors cover 80-90% of practical scraping needs with simpler, more readable syntax.

When to use XPath

Use XPath when:

You need parent traversal:

# Select the div containing a price element (CSS cannot go to parent)
//span[@class='price']/ancestor::div[@class='product-card']

You need text-based element matching:

# Find button that contains the text "Add to Cart"
//button[contains(text(), 'Add to Cart')]

You need following-sibling or preceding-sibling traversal:

# Select the paragraph immediately after a specific h2
//h2[@class='specifications-heading']/following-sibling::p[1]

The HTML uses no useful classes or IDs (rare but exists in legacy sites):

# Select by document position (fragile but sometimes the only option)
/html/body/div[3]/table/tr[2]/td[1]

Practical comparison

Task	CSS	XPath
Select by class	.product-title	//[contains(@class, 'product-title')]
Select text content	h1::text	//h1/text()
Select attribute	a::attr(href)	//a/@href
Parent element	Not possible	//span/parent::div
Text matching	Not possible	//button[text()='Submit']
Position-based	div:nth-child(2)	//div[2]
Regex matching	Not possible	//p[matches(text(), '^Price:')] (XPath 2.0)

Code in different frameworks

Beautiful Soup:

# CSS selector
soup.select_one("div.product-card h2.name")
soup.select("div.product-card")

# XPath via lxml parser
from lxml import html
tree = html.fromstring(response.text)
elements = tree.xpath("//div[contains(@class, 'product-card')]//h2")

Scrapy:

# Both syntaxes are first-class
response.css("div.product-card h2::text").get()
response.xpath("//div[contains(@class, 'product-card')]//h2/text()").get()

Playwright:

# CSS selector (preferred in Playwright)
page.query_selector(".product-card h2")

# XPath
page.query_selector("xpath=//div[contains(@class, 'product-card')]//h2")

The fragility problem

Both CSS selectors and XPath expressions tied to specific HTML structure break when the page layout changes. Positional XPath (/html/body/div[3]/td[1]) breaks on any layout change. Class-based CSS (div.price) breaks when the class name changes.

More resilient selectors use semantic cues:

• [data-testid="product-price"] — test IDs are more stable than visual class names
• [aria-label="product price"] — accessibility labels are often stable
• //span[@itemprop='price'] — structured data markup is very stable

When scraping a site you will maintain long-term, invest time in identifying the most stable selectors available. Data attributes and schema.org markup are your friends.