Web Scraping Glossary

60 terms, developer-level definitions. Every entry cross-links to relevant tool reviews, comparisons, and how-to guides. No fluff.

A

Anti-bot

Anti-bot detection

Anti-bot detection systems identify and block automated web scraping traffic using IP reputation, browser fingerprinting, behavioral analysis, and CAPTCHA challenges.

→ Definition

C

Anti-bot

CAPTCHA solver

A CAPTCHA solver is a service or technique that automatically completes CAPTCHA challenges during web scraping — either through AI image recognition, audio transcription, or human worker farms.

→ Definition

H

Web scraping tools

Headless Browser

A headless browser is a full web browser (Chrome, Firefox) that runs without a visible user interface. It executes JavaScript, renders CSS, handles cookies and sessions, and produces a complete DOM — making it essential for scraping JavaScript-rendered pages.

→ Definition

Web scraping fundamentals

HTTP Status Codes

HTTP status codes are three-digit numbers returned by web servers indicating the result of each request. For web scrapers, certain status codes signal blocks, rate limits, or access restrictions that require specific handling strategies.

→ Definition

J

Web scraping techniques

JSON API Scraping

JSON API scraping extracts data directly from a website's underlying API endpoints rather than from rendered HTML. It is significantly faster and more reliable than HTML scraping when the site's XHR/fetch calls are identifiable in browser DevTools.

→ Definition

P

Web scraping techniques

Pagination Handling

Pagination handling is the technique of following page-by-page navigation in web scrapers. It covers URL-based pagination, next-button pagination, infinite scroll, and load-more button patterns — each requiring a different implementation approach.

→ Definition

Anti-detection techniques

Proxy Rotation

Proxy rotation changes the IP address used for each request or session during web scraping, preventing rate limiting and IP bans that target a single source IP. Residential proxy rotation uses IPs from real consumer devices, making detection significantly harder.

→ Definition

R

Web scraping fundamentals

Rate Limiting

Rate limiting is a server-side mechanism that restricts how frequently a client can make requests. Web scrapers that ignore rate limits trigger HTTP 429 (Too Many Requests) responses and eventually IP bans.

→ Definition

Proxy

Residential proxy

A residential proxy routes your requests through an IP address assigned by an ISP to a real home or mobile device — making your scraper look like a regular user, not a datacenter bot.

→ Definition

Web scraping fundamentals

Robots.txt

robots.txt is a text file at the root of a website that communicates crawling permissions to web crawlers and bots. It is not legally binding but has significant implications for scraping legality, ethics, and terms of service compliance.

→ Definition

Proxy

Rotating proxy

A rotating proxy automatically changes the IP address used for each HTTP request (or each session), distributing scraping traffic across a pool of IPs to avoid rate-limiting and IP bans.

→ Definition

S

Web scraping tools

Selenium

Selenium is the original browser automation framework, widely used for web testing and scraping JavaScript-rendered pages. It is slower and heavier than Playwright but has broader language support and a larger existing codebase. In 2025, most new scraping projects choose Playwright over Selenium.

→ Definition

U

Web scraping fundamentals

User-Agent

The User-Agent is an HTTP request header that identifies the client making the request. Web browsers send descriptive strings; scraping libraries send identifiable strings by default. Setting a realistic User-Agent is the first step in avoiding bot detection.

→ Definition

W

Tools

Web scraping API

A web scraping API is a managed HTTP service that handles proxy rotation, CAPTCHA solving, and browser rendering on your behalf — you send a URL, it returns the page HTML or structured data.

→ Definition

X

Web scraping fundamentals

XPath vs CSS Selectors

XPath and CSS selectors are two ways to locate elements in an HTML document. CSS selectors are simpler and sufficient for most scraping; XPath is more powerful for complex traversal patterns, text-based selection, and navigating non-standard HTML structures.

→ Definition