Pagination Handling

The four pagination patterns

Most websites paginate content in one of four ways. Each requires a different scraping approach.

Pattern 1: URL-based pagination

The page number appears in the URL:

• https://example.com/products?page=1
• https://example.com/products/page/2
• https://example.com/products?offset=100&limit=50

This is the easiest to handle — generate all URLs programmatically:

# Scrapy
def start_requests(self):
    for page in range(1, 101):  # Pages 1-100
        yield scrapy.Request(
            f"https://example.com/products?page={page}",
            callback=self.parse
        )

# requests (with unknown total pages — find last page first)
def get_all_pages(base_url):
    first_response = requests.get(f"{base_url}?page=1")
    soup = BeautifulSoup(first_response.text, "html.parser")
    
    # Find total pages from pagination links
    last_page_link = soup.select_one("a.pagination-last")
    total_pages = int(last_page_link["href"].split("page=")[1]) if last_page_link else 1
    
    for page in range(1, total_pages + 1):
        yield f"{base_url}?page={page}"

Pattern 2: Next-button pagination

Each page has a “Next” or “Next page” link pointing to the next URL:

<a class="next-page" href="/products?page=3">Next</a>

Follow the chain:

# Beautiful Soup
url = "https://example.com/products"
while url:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    
    # Extract items on this page
    for item in soup.select(".product-card"):
        yield parse_item(item)
    
    # Find next page
    next_link = soup.select_one("a.next-page")
    url = "https://example.com" + next_link["href"] if next_link else None
    
    time.sleep(1)

# Scrapy
def parse(self, response):
    for item in response.css(".product-card"):
        yield parse_item(item)
    
    next_page = response.css("a.next-page::attr(href)").get()
    if next_page:
        yield response.follow(next_page, self.parse)

Pattern 3: Infinite scroll

New content loads as the user scrolls down. Requires a headless browser:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")
    
    previous_count = 0
    while True:
        # Scroll to bottom
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        page.wait_for_timeout(2000)
        
        current_count = len(page.query_selector_all(".product-card"))
        if current_count == previous_count:
            break  # No new items loaded
        previous_count = current_count
    
    products = page.query_selector_all(".product-card")
    browser.close()

API interception approach (faster): Many infinite-scroll sites load data via a paginated API. Instead of controlling scroll, intercept the network requests and call the API directly.

Pattern 4: Load More button

A button triggers loading of additional items without changing the URL:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products")
    
    while True:
        load_more = page.query_selector("button.load-more")
        if not load_more or not load_more.is_visible():
            break
        
        previous_count = len(page.query_selector_all(".product-card"))
        load_more.click()
        
        # Wait for new items to appear
        page.wait_for_function(
            f"document.querySelectorAll('.product-card').length > {previous_count}",
            timeout=5000
        )
    
    products = page.query_selector_all(".product-card")
    browser.close()

Detecting which pattern a site uses

1. Load the target page in Chrome
2. Scroll to the bottom — if content loads automatically, it’s infinite scroll
3. Look for a button labeled “Load More,” “Show More,” or “See More” — load-more button pattern
4. Look at the URL when you click to the second page — if it changes, it’s URL-based
5. Look for a “Next” or “2” link in the pagination footer — next-button pattern

Pagination discovery in Scrapy (CrawlSpider)

Scrapy’s CrawlSpider can automatically follow pagination links matching a pattern:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ProductSpider(CrawlSpider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/products"]
    
    rules = (
        # Follow pagination links
        Rule(LinkExtractor(restrict_css="a.next-page"), callback="parse_products", follow=True),
    )
    
    def parse_products(self, response):
        for item in response.css(".product-card"):
            yield {"name": item.css("h2::text").get()}

Handling duplicate pages

URL-based pagination with inconsistent parameters can produce duplicate pages:

• /products?page=1&sort=price
• /products?sort=price&page=1

These are the same page with different parameter order. Scrapy’s duplicate filter (enabled by default via DUPEFILTER_CLASS) handles this for Scrapy spiders. For custom scrapers, maintain a visited-URL set:

visited = set()

def scrape_page(url):
    normalized_url = normalize_url(url)  # Sort params, lowercase
    if normalized_url in visited:
        return
    visited.add(normalized_url)
    # ... scrape