Web Scraping Use Cases by Industry: What Data People Actually Collect

Why use case matters for tool choice

The right scraping stack for monitoring competitor prices on 50,000 product pages is different from the right stack for collecting real estate listings once per day. The right stack for extracting structured data from financial statements is different from collecting social media profiles.

This article maps common industry use cases to the technical approaches and tools that actually work for them, based on the properties of each target.

E-commerce: price monitoring and product data

What’s collected: Product names, SKUs, prices, availability, ratings, review counts, seller information, shipping options.

Scale: Typically 10K–10M product pages per day for competitive intelligence.

Technical approach: Most e-commerce sites render product data in HTML (not JS-dependent) because they want Google to crawl it. Scrapy with XPath or CSS selectors handles this efficiently.

Amazon is the exception — it has some of the most aggressive bot detection in e-commerce, with Cloudflare protection, CAPTCHA challenges, and rate limiting. At scale, Amazon scraping typically requires:

• Residential proxy rotation (Bright Data or Smartproxy)
• Playwright-stealth for session management
• Or a pre-built Amazon scraper (Apify’s Amazon Product Scraper has high reliability)

Typical data pipeline:

1. Scrapy spider with custom downloader middleware for proxy rotation
2. Item pipelines for data cleaning and deduplication
3. Output to PostgreSQL or BigQuery for price trend analysis
4. Scheduled via Apify Cloud or Cron + Scrapy Cloud

Target complexity examples:

• Amazon product pages: High protection → residential proxies + stealth browser required
• Shopify stores: Low-moderate protection → Scrapy + datacenter proxies sufficient
• WooCommerce stores: Low protection → basic requests often works

Real estate: property listings

What’s collected: Property addresses, prices, square footage, bedroom/bathroom counts, listing dates, photos, agent information, price history.

Scale: Moderate — thousands of new listings per day, but total market data is millions of historical records.

Target complexity: Rightmove, Zoopla, and OnTheMarket (UK market) all have anti-bot measures but they need Google to crawl them, which limits how aggressive they can be. A well-configured Scrapy spider with residential proxies and proper delay settings handles these at moderate scale.

The property ID pattern: Most real estate portals use sequential or predictable property IDs in URLs (/property/123456). Rather than crawling through listing pages, you can build a URL set and hit property pages directly — more efficient and less detectable than crawling listing pages.

Special considerations: Property data contains addresses and sometimes owner information. In the UK, the Land Registry provides official property sales data as an open dataset (no scraping needed for historical transactions). Scraping is more valuable for current listings and agent information.

Job boards: labour market intelligence

What’s collected: Job titles, company names, locations, salaries (when shown), required skills, experience level, posting dates, employer size.

Use cases:

• Salary benchmarking (what does a Senior React Developer earn in Manchester?)
• Skills demand tracking (is Python growing relative to Java in job postings?)
• Recruiting automation (finding companies that are actively hiring)
• Market research (which sectors are growing or contracting?)

Technical approach: Reed.co.uk, Indeed, and Totaljobs are the main UK job boards. Each has API terms of service — Reed has a legitimate public API; Indeed restricts scraping in their ToS. For research and benchmarking (not automated job applications), scraping is common.

Indeed’s search results are server-side rendered but the site uses Cloudflare. Scrapy + rotating proxies handles low-volume collection; at scale, consider Indeed’s official Hiring API or a scraping service.

Better approach for job data at scale: LinkedIn’s Official Talent Insights product, Indeed’s Data API, and Lightcast (formerly EMSI Burning Glass) all sell structured job posting data. These are worth comparing against DIY scraping costs at volumes above 100K postings/month.

Lead generation: contact and company data

What’s collected: Business names, contact names, email addresses (when public), phone numbers, websites, LinkedIn profiles, addresses.

Legal caution: This is the highest-risk category from a UK GDPR perspective. Scraping personal contact information (especially email addresses) for commercial outreach without a legitimate basis or consent creates significant legal exposure. B2B contact information for business contacts has somewhat clearer legal grounding under “legitimate interests” — but only when processed appropriately and without ignoring opt-out signals.

Common sources:

• Companies House (official UK company registry — free, well-structured API, no scraping needed)
• Google Maps (local business data — Google’s ToS prohibits scraping but the Maps scraping use case is extremely common)
• LinkedIn (restrictive ToS, hiQ v LinkedIn established US legal precedent for public data, but UK law is different)
• Company websites’ “About” or “Team” pages

Technical approach for Google Maps: Playwright automation is required (Maps is fully JS-rendered). Search queries, pan the map, extract business card data. Rate limiting is critical — Google Maps blocks IP ranges quickly. Apify’s Google Maps Scraper (community Actor) is a common tool for this use case.

For B2B lead generation at scale, evaluate Apollo.io, Hunter.io, and ZoomInfo as data purchasers vs. scraping cost — they often have coverage advantages and clearer legal standing for B2B outreach.

Financial data: prices, filings, and earnings

What’s collected: Stock prices, earnings reports, balance sheet items, regulatory filings, fund holdings, option chains, analyst estimates.

Official sources first: Before scraping financial data, check official sources:

• Companies House: UK company filings (free, structured)
• SEC EDGAR: US company filings (free API)
• LSE/FCA Regulatory News Service: UK regulatory announcements
• OpenFIGI, ISIN.org: Security identifiers
• Yahoo Finance Unofficial API, Alpha Vantage (free tier): Price data

Many financial data use cases are better served by these official APIs than by scraping.

When scraping is necessary:

• Proprietary or aggregated data not available via API (specific screener data, unusual metrics)
• Real-time price data not available in free APIs
• Alternative data (shipping rates, job postings as economic signals)

Technical complexity: Financial data sites (Bloomberg, Refinitiv, FactSet) have enterprise access controls and serious bot protection. Scraping these at meaningful scale requires residential proxies and is explicitly prohibited in most terms of service. EODHD, Polygon.io, and Tiingo provide similar data via legitimate APIs at reasonable prices.

News and media monitoring

What’s collected: Article text, publication dates, authors, categories, sentiment, citation links.

Technical approach: News sites are generally well-structured HTML (optimised for Google). RSS feeds are the correct first approach — many news sites publish RSS for their categories. For sites without RSS, Scrapy handles most news content efficiently.

Common patterns:

• Sitemap crawling (news sitemaps at /news-sitemap.xml list recent articles)
• RSS polling every 15 minutes for breaking news
• Full-text extraction with newspaper3k or trafilatura (which handles boilerplate removal better than generic extractors)

Scale: A news monitoring system covering 50 UK news sources can collect and process all articles published in a day in under 30 minutes with a single Scrapy instance. This is a low-complexity use case technically.

Attribution and copyright: Collecting article text for analysis (sentiment analysis, trend monitoring) is more defensible legally than republishing it. Automated republication of news content is a copyright issue regardless of how it’s collected.

Choosing your approach by use case