Web Scraping at Scale: Challenges and Solutions

Collecting public data at scale is mostly an engineering problem: how to stay reliable as volume grows while remaining a good citizen of the sites you visit.

This guide follows a job from planning to monitoring: the rules and checks, a request budget, a polite fetcher with retries, proxies, validation, storage and the numbers worth watching. The code is Python; replace the placeholders with your own details.

1.Plan before you scale

Define exactly which public data you need, check the website's terms and robots.txt, and decide how often the data really needs to be refreshed. Collecting less, less often, is the cheapest way to scale.

Answer these questions before the first request:

1. Is the data public, and may you use it? Read the terms of use. Copyright, database rights and the rules on personal data can apply even to public pages.
2. Is there an API, a feed or a sitemap? Official data endpoints are lighter, steadier and welcome. Use them first.
3. Which fields do you need, and how fresh must they be? A daily refresh costs a fraction of an hourly one.
4. What is your budget? Fix the requests per day, the GB per month and the run time you can afford.
5. How will the site reach you? Identify your crawler with a clear User-Agent and a contact address.

2.Check the rules first

Read robots.txt before you crawl and follow it. The standard library can parse it, including a Crawl-delay when the site sets one. Download the file with your own session, so the request also goes through your proxy:

from urllib.robotparser import RobotFileParser
import requests

USER_AGENT = "my-crawler/1.0 (+https://example.com/crawler; contact: [email protected])"

proxy = "http://USERNAME:PASSWORD@HOST:PORT"
session = requests.Session()
session.proxies.update({"http": proxy, "https": proxy})
session.headers["User-Agent"] = USER_AGENT

def load_robots(base):
    r = session.get(base + "/robots.txt", timeout=(5, 20))
    robots = RobotFileParser()
    robots.parse(r.text.splitlines() if r.ok else [])      # no robots.txt: nothing is disallowed
    return robots

robots = load_robots("https://example.com")
delay = robots.crawl_delay("my-crawler") or 2               # the site's own value, or 2 seconds
print(robots.can_fetch("my-crawler", "https://example.com/page/1"), delay)

Personal data and the law

A public page can still contain personal data, and rules such as the GDPR may apply to it. Collect only the fields you need, say why you collect them, keep them no longer than necessary and delete them on request where the law requires it. This is general information, not legal advice: ask a lawyer when your project involves people’s data.

3.Control your request rate

Set a global request budget and per-site limits. Add delays between requests, limit concurrency and slow down when a site responds slowly or returns errors.

The arithmetic is simple. At one request every 2 seconds per site, 10,000 pages take about 5.5 hours. If that is too slow, reduce the number of pages or ask the site owner about an official feed, rather than raising the rate. This throttle keeps a minimum gap per host, with a little random jitter so that workers do not fire in lockstep:

import random
import threading
import time
from urllib.parse import urlsplit

class Throttle:
    """At least <delay> seconds between two requests to the same host."""

    def __init__(self, delay):
        self.delay = delay
        self.next_at = {}
        self.lock = threading.Lock()

    def wait(self, url):
        host = urlsplit(url).netloc
        with self.lock:
            now = time.monotonic()
            at = max(self.next_at.get(host, now), now)
            self.next_at[host] = at + self.delay + random.uniform(0, self.delay * 0.25)
        time.sleep(at - now)

throttle = Throttle(delay)

Also ask the site whether a page changed instead of downloading it again. Servers that send an ETag answer 304 Not Modified with no body, which saves time, GB and load on the site:

cache = {}        # url -> {"etag": ..., "body": ...}

def get_cached(url):
    headers = {}
    if url in cache and cache[url]["etag"]:
        headers["If-None-Match"] = cache[url]["etag"]
    r = session.get(url, headers=headers, timeout=(5, 20))
    if r.status_code == 304:
        return cache[url]["body"]                # unchanged: use the copy you already have
    r.raise_for_status()
    cache[url] = {"etag": r.headers.get("ETag"), "body": r.text}
    return r.text

4.Handle errors gracefully

Use timeouts, retry transient errors with exponential backoff and stop retrying permanent failures. Log every failure with enough context to understand it later.

This function applies those rules. It waits longer after every failed attempt, honors Retry-After and stops at once when access is refused:

def fetch(session, url, attempts=4):
    for attempt in range(attempts):
        throttle.wait(url)
        try:
            r = session.get(url, timeout=(5, 20))
        except requests.exceptions.RequestException:
            r = None                                          # connection problem
        if r is not None:
            if r.status_code < 400:
                return r
            if r.status_code in (404, 410):
                return r                                      # gone: nothing to retry
            if r.status_code in (401, 403, 407):
                raise PermissionError(f"{r.status_code} for {url}: stop and check the rules and your credentials")
        pause = 2 ** attempt + random.uniform(0, 1)            # 1 s, 2 s, 4 s ... plus jitter
        if r is not None and r.headers.get("Retry-After", "").isdigit():
            pause = max(pause, int(r.headers["Retry-After"]))  # the site says how long to wait
        time.sleep(pause)
    return None

A circuit breaker protects the site and your budget. When too many of the latest requests fail, stop the crawl of that host, wait and look at the cause. Pushing harder rarely helps:

from collections import deque

class CircuitBreaker:
    """Opens when too many of the latest requests failed."""

    def __init__(self, window=20, max_failures=10):
        self.results = deque(maxlen=window)
        self.max_failures = max_failures

    def record(self, ok):
        self.results.append(ok)

    @property
    def open(self):
        return list(self.results).count(False) >= self.max_failures

5.Use sessions and rotation wisely

Change the IP on demand between independent jobs and keep the same IP for workflows that span several requests. Reuse connections where you can to reduce overhead.

Several proxies spread a workload across your connections. They do not change the site’s rules: the throttle above works per host, so the whole crawl stays within the limit however many proxies you use. Give every worker thread its own session and proxy, and read how to rotate proxies for pools, cooldowns and per-proxy pacing. This crawl ties the pieces together:

import itertools
from concurrent.futures import ThreadPoolExecutor

proxy_urls = ["http://USER1:PASS1@HOST1:PORT1", "http://USER2:PASS2@HOST2:PORT2"]
turns = itertools.cycle(proxy_urls)
local = threading.local()
breaker = CircuitBreaker()

def thread_session():
    if not hasattr(local, "session"):
        s = requests.Session()
        proxy = next(turns)                                   # one proxy per worker thread
        s.proxies.update({"http": proxy, "https": proxy})
        s.headers["User-Agent"] = USER_AGENT
        local.session = s
    return local.session

def crawl(url):
    if breaker.open:
        return None                                           # stop quietly: investigate first
    try:
        r = fetch(thread_session(), url)
    except PermissionError as exc:
        print(exc)
        r = None
    breaker.record(r is not None)
    return r

urls = [f"https://example.com/page/{i}" for i in range(1, 9)]

with ThreadPoolExecutor(max_workers=2) as pool:               # no more workers than your plan allows
    results = list(pool.map(crawl, urls))

print(sum(r is not None for r in results), "of", len(urls), "pages fetched")

6.Store and validate data

Validate each record against a schema, remove duplicates and monitor the share of empty or malformed results. A sudden change usually means the target page changed.

from dataclasses import dataclass

@dataclass
class Product:
    sku: str
    title: str
    price: float

def validate(row):
    try:
        product = Product(row["sku"].strip(), row["title"].strip(), float(row["price"]))
    except (KeyError, ValueError, AttributeError):
        return None
    if not product.sku or not product.title or product.price <= 0:
        return None
    return product

rows = [{"sku": "A1", "title": "Phone case", "price": "9.90"}, {"sku": "", "title": "?", "price": "-1"}]
valid = [p for p in map(validate, rows) if p]

invalid_share = 1 - len(valid) / max(len(rows), 1)
if invalid_share > 0.05:
    print(f"{invalid_share:.0%} of the rows failed validation: the page may have changed")

Store the data in a way that makes a re-run safe. An upsert on a stable key updates what changed and creates no duplicates, and SQLite in the standard library is enough for many jobs:

import sqlite3

db = sqlite3.connect("products.db")
db.execute("CREATE TABLE IF NOT EXISTS products (sku TEXT PRIMARY KEY, title TEXT, price REAL, seen_at TEXT)")

def save(p):
    db.execute(
        "INSERT INTO products VALUES (?, ?, ?, datetime('now')) "
        "ON CONFLICT(sku) DO UPDATE SET title = excluded.title, price = excluded.price, seen_at = excluded.seen_at",
        (p.sku, p.title, p.price),
    )
    db.commit()

for product in valid:
    save(product)

Keep the raw responses for a while as well. When a parser breaks, you can fix it and re-run on the saved pages instead of downloading them again.

7.Watch the numbers

A crawl you cannot see is a crawl you cannot trust. Count what happens and print a summary at the end of every run:

from collections import Counter

stats = Counter()

def count(r):
    stats["requests"] += 1
    if r is None:
        stats["failed"] += 1
        return
    stats[f"status_{r.status_code}"] += 1
    stats["bytes"] += len(r.content)

for r in results:
    count(r)

print(dict(stats), f"{stats['bytes'] / 1024**2:.2f} MB")

Watch five numbers: the share of failed requests, the share of 429 responses, the average response time, the GB used against your budget and the share of invalid records. When one of them jumps, pause the crawl and find out why before you continue.

8.Common mistakes

• Starting at full speed. Begin slowly and raise the rate only while the error rate stays flat and the site’s rules allow it.
• Answering 403 and 429 with more proxies. Those responses ask you to stop or slow down. Review the terms, lower the rate or contact the site owner.
• Ignoring robots.txt and the terms. Read them before the crawl, not after a complaint.
• Retrying everything. Retry transient errors only, with a backoff and a limit.
• Parsing without validation. A changed page then fills your database with empty or wrong values.
• Fetching full pages when a data endpoint exists. It wastes GB and loads the site for nothing.

9.FAQs

Is web scraping legal?

It depends on the data, the website's terms and applicable law. Collect only publicly available data you are entitled to use, and respect robots.txt and rate limits.

How fast can I crawl?

As fast as the site allows. Use its Crawl-delay when it sets one, start with one request every few seconds per host and slow down at the first signs of strain.

Do I need a headless browser?

Only when the data appears after JavaScript runs. Plain HTTP requests are faster and lighter in GB, and an official API or data endpoint is better still.

How many proxies do I need?

Divide the requests per minute you need by what one proxy can safely send within the site’s limits, and start small. How to rotate proxies has a worked example.

How do I keep the traffic bill down?

Use data endpoints instead of full pages, send conditional requests, skip images and fonts in browsers, cache what does not change and collect less often. Mobile vs datacenter proxies shows how to estimate GB.

Still have questions about mobile proxies? Visit our FAQ page for detailed answers, or contact our support team — we're here to help.