Structured Data Extraction from HTML with LLMs: Using GPT-4o and Claude

Why LLMs for data extraction?

CSS selectors and XPath work reliably when HTML structure is consistent. They break when:

• The same data appears in different HTML structures across pages (common on product detail pages from different vendors)
• Text fields have inconsistent formats (“£29.99”, “GBP 29.99”, “29 pounds 99 pence”)
• Data is embedded in paragraph text rather than structured elements
• Page layouts vary by region, device, or A/B test

LLMs excel precisely where selectors fail: understanding natural language, handling format variation, and extracting meaning from unstructured text.

The basic extraction pattern

The core pattern: fetch the HTML, extract the relevant text, pass it to an LLM with a structured output instruction.

import anthropic
import json
import requests
from bs4 import BeautifulSoup

client = anthropic.Anthropic()

def extract_product_data(url: str) -> dict:
    # Fetch and clean the HTML
    response = requests.get(url, headers={"User-Agent": "Mozilla/5.0 ..."})
    soup = BeautifulSoup(response.text, "html.parser")
    
    # Remove noise: scripts, styles, nav, footer
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    
    # Get clean text (much cheaper than sending full HTML)
    page_text = soup.get_text(separator="\n", strip=True)
    
    # Truncate to relevant portion (most products are in first 2000 chars of clean text)
    relevant_text = page_text[:3000]
    
    # Extract with Claude
    message = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=500,
        messages=[
            {
                "role": "user",
                "content": f"""Extract structured product data from this page text.
Return ONLY valid JSON with these fields: name, price_gbp (number), sku, in_stock (boolean), rating (number or null).
If a field cannot be found, use null.

Page text:
{relevant_text}"""
            }
        ]
    )
    
    response_text = message.content[0].text.strip()
    # Remove markdown code blocks if present
    if response_text.startswith("```"):
        response_text = response_text.split("```")[1]
        if response_text.startswith("json"):
            response_text = response_text[4:]
    
    return json.loads(response_text)

# Usage
product = extract_product_data("https://example-store.com/product/widget-42")
print(product)
# {"name": "Premium Widget 42", "price_gbp": 29.99, "sku": "WGT-042", "in_stock": true, "rating": 4.5}

Using GPT-4o with structured outputs

OpenAI’s GPT-4o with JSON mode guarantees valid JSON output:

from openai import OpenAI
from pydantic import BaseModel
import requests
from bs4 import BeautifulSoup

client = OpenAI()

class ProductData(BaseModel):
    name: str
    price_gbp: float | None
    sku: str | None
    in_stock: bool
    rating: float | None
    description: str | None

def extract_with_gpt4o(page_text: str) -> ProductData:
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Extract structured product data from the provided page text."},
            {"role": "user", "content": page_text[:4000]}
        ],
        response_format=ProductData,
    )
    return response.choices[0].message.parsed

OpenAI’s Structured Outputs mode with Pydantic ensures the response matches your schema exactly — no JSON parsing errors.

Cost analysis (2026 pricing)

For a scraping project processing 10,000 product pages:

Claude claude-sonnet-4-6 (claude-sonnet-4-6):

• Input: ~1,000 tokens/page (cleaned text) × 10,000 = 10M tokens
• Output: ~200 tokens/page × 10,000 = 2M tokens
• Cost: $3/M input + $15/M output = $30 + $30 = $60 total

GPT-4o:

• Same token counts
• Cost: $5/M input + $15/M output = $50 + $30 = $80 total

CSS selector-based extraction (no LLM):

• Zero API cost; compute cost negligible
• $0 total

LLMs cost $60-80 per 10,000 pages. CSS selectors cost $0. The LLM investment is only worth it when:

• CSS selectors are unreliable (inconsistent HTML)
• The time to maintain selectors as layouts change exceeds $60 per maintenance cycle
• The quality improvement justifies the cost

When LLMs clearly beat CSS selectors

Multi-source aggregation: You are scraping 200 different e-commerce sites, each with different HTML structure. Writing 200 custom selectors is impractical. One LLM prompt works across all of them.

Free-text fields with format variation: Prices written as “$29.99”, “USD 29.99”, “Twenty-nine dollars and ninety-nine cents”, or “29.99 USD”. A selector can extract the text; an LLM can parse any format into a float.

Extracting from paragraph text: “Delivery typically takes 3-5 business days” — no selector can reliably extract “3-5 business days” from this. An LLM understands it.

Tables without column headers: Some legacy sites have data in tables with no headers. An LLM can understand the context (“this appears to be a technical specifications table”) where a selector just sees rows and cells.

Implicit data: “Our products ship from the UK” → ships_from: "UK" — an LLM can infer this from natural language; a selector cannot.

When LLMs are not worth it

Consistent HTML structure: If every product page on a site uses the same template, a Scrapy spider with CSS selectors is 100x cheaper. Use selectors.

Very high volume: At 10M pages/month, LLM costs become $60,000-$80,000/month. Unless the data quality improvement justifies this, use selectors or hybrid approaches.

Real-time requirements: LLM API calls add 0.5-3 seconds per page. For applications requiring fresh data at high speed, this latency is unacceptable.

Hybrid extraction: selectors + LLM fallback

The most practical architecture for production use:

def extract_product(url: str) -> dict:
    response = requests.get(url, headers=get_headers())
    soup = BeautifulSoup(response.text, "html.parser")
    
    # Try CSS selector first (fast, free)
    name = soup.select_one("h1.product-title")
    price = soup.select_one("span.price")
    
    if name and price:
        # Selector-based extraction succeeded
        return {
            "name": name.get_text(strip=True),
            "price": parse_price(price.get_text(strip=True))
        }
    
    # Selector failed — fall back to LLM
    page_text = soup.get_text(separator="\n", strip=True)[:3000]
    return extract_with_llm(page_text)

This pattern uses free selectors for the majority of pages (consistent layouts) and LLM fallback only for unusual cases. Cost drops significantly compared to LLM-only extraction.

Reducing token costs

LLM pricing is per token. Sending full HTML to an LLM is expensive — most HTML is noise.

Pre-processing to reduce tokens:

def clean_for_llm(html: str, max_chars: int = 3000) -> str:
    soup = BeautifulSoup(html, "html.parser")
    
    # Remove structural noise
    for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
        tag.decompose()
    
    # Get clean text
    text = soup.get_text(separator="\n", strip=True)
    
    # Remove consecutive blank lines
    import re
    text = re.sub(r'\n{3,}', '\n\n', text)
    
    return text[:max_chars]

A 100KB HTML page typically reduces to 2,000-4,000 characters of relevant text — a 10-30x token reduction.

Prompt engineering for extraction

Be specific about the output format: “Return ONLY valid JSON” reduces the risk of the LLM returning explanation text alongside the JSON.

Provide examples in the prompt (few-shot):

Example input text: "Widget Pro — £49.99 — In Stock — SKU: WP-001"
Example output: {"name": "Widget Pro", "price_gbp": 49.99, "in_stock": true, "sku": "WP-001"}

Now extract from: [actual page text]

Specify null handling: “If a field cannot be found, use null rather than guessing.” This prevents hallucination of data that is not present.

Validate outputs: Always validate LLM-extracted data with type checks and range checks. A price field returning a string instead of a float, or a rating field returning 47.5 instead of 4.75, should trigger a re-extraction or manual review flag.