Web Scraping for AI Training Data and RAG, Done Responsibly

Large language models and retrieval-augmented generation (RAG) both start with text from the web. Collecting it is easy. Collecting it responsibly takes decisions made before the first request: which sources, under what terms, how fast, and how you will show later where every document came from.

This guide separates RAG from training, lists the rules that decide what you may collect and walks through a pipeline: check the rules, fetch politely, clean, deduplicate, record provenance, store and delete. It is general information and not legal advice. For a real dataset, ask a lawyer who knows copyright and data protection in your markets.

1.RAG and training are different jobs

The same crawler can feed both, but the two uses carry different obligations. Decide which one you are building before you collect anything.

Because removal is easy in an index and hard in a trained model, many teams start with RAG and keep the source of every chunk. The rest of this guide applies to both.

2.The rules that decide what you may collect

There is no single rule. Several things apply at once, and the strictest one wins:

• robots.txt. A convention (RFC 9309) with which site owners tell crawlers which paths are open. It is not a law by itself, but ignoring it is a weak position in any dispute. Many sites now name AI crawlers in it; tokens such as GPTBot, ClaudeBot, CCBot and Google-Extended are examples.
• Terms of use. A contract. Some sites forbid automated collection or the use of their content for AI training.
• Copyright and licences. Pages are protected works unless they carry an open licence (for example Creative Commons) or are in the public domain. In the EU, the text and data mining rules (Article 4 of Directive 2019/790) let rights holders reserve their rights in a machine-readable way, and a robots.txt entry is commonly read as one such signal. The details are still being settled, so treat an opt-out as binding.
• Personal data. Names, e-mail addresses and profiles stay personal data when they are public. GDPR asks for a legal basis, data minimisation and deletion on request.
• Access controls. Pages behind a login or a paywall are outside this guide. Collect them only with permission.

When in doubt, prefer sources that give you a licence: open data, official APIs, partner feeds and content you own. Our Acceptable Use Policy lists what the service may not be used for, and secure and responsible proxy use covers the habits around it.

3.A responsible pipeline at a glance

1. Rules. Read robots.txt, the terms and the licence of each source, and write down the decision.
2. Fetch. Request pages slowly with an honest User-Agent, and only what you need.
3. Clean. Keep the content, drop menus, pop-ups and scripts.
4. Deduplicate. Remove exact and near copies.
5. Store. Keep the text with its provenance, a retention period and a way to delete.

4.Check the rules before the first request

Fetch robots.txt once per site, through the same session and proxy you will crawl with, and ask the parser for every URL before you request it. A missing file (404) means nothing is disallowed. A server error means the site is in trouble, so wait and try later:

from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser

import requests

UA = "my-crawler/1.0 (+https://example.com/crawler; contact: [email protected])"
_robots: dict[str, RobotFileParser] = {}

def allowed(session: requests.Session, url: str) -> bool:
    parts = urlsplit(url)
    base = f"{parts.scheme}://{parts.netloc}"
    if base not in _robots:
        r = session.get(base + "/robots.txt", timeout=(5, 20))
        if r.status_code >= 500:
            return False                        # the site is in trouble: try again later
        rp = RobotFileParser()
        rp.parse(r.text.splitlines() if r.status_code == 200 else [])
        _robots[base] = rp
    return _robots[base].can_fetch(UA, url)

The parser in the standard library is simple and does not implement every part of RFC 9309, such as wildcards, so use a stricter parser for a large crawl. Also read the meta robots tag and the X-Robots-Tag header of the pages you fetch, and the terms of use of the site. Save the date and a hash of the robots.txt you read: it is part of your proof.

5.Fetch politely

Politeness is also what keeps a crawl running for months. Send an honest User-Agent with a page that describes your crawler and a contact address, keep one request in flight per site, pause between requests, and slow down when the site asks. Conditional requests save traffic too: a 304 answer has no body.

import random
import time

def retry_after(r) -> int:
    value = r.headers.get("Retry-After", "")
    return int(value) if value.isdigit() else 60

def polite_get(session, url, etag=None, pause=(2, 5)):
    if not allowed(session, url):
        return None                             # the site asks crawlers to stay out
    headers = {"If-None-Match": etag} if etag else {}
    r = session.get(url, headers=headers, timeout=(5, 30))
    if r.status_code in (429, 503):
        time.sleep(retry_after(r))              # the site asked us to slow down
        return None
    time.sleep(random.uniform(*pause))          # a pause between requests to the same site
    return r                                    # status 304 means: unchanged, keep the stored copy

Request budgets, error handling and validation are covered in web scraping at scale, and the Python proxies guide shows sessions, timeouts and retries.

6.Where proxies fit in the pipeline

Proxies are one tool in the fetch step. They are useful when the public page you need depends on the country: local prices, product availability, regulatory notices, documentation for one market. A dataset or a RAG index for a market should hold the page that market sees.

• One IP per crawl job. With Xameleon the IP stays the same while you work. Change it between jobs, from your account or through the API, and log the country of every fetch. How to rotate proxies shows the patterns.
• Pick the type for the page. Public documentation and open data usually work from datacenter proxies. Pages that are tailored to mobile networks fit 4G mobile proxies. Mobile versus datacenter proxies compares them.
• Keep the limit per site, not per IP. Ten proxies are not permission for ten times the load.

A proxy changes where a request comes from. It does not change whether you are allowed to make it: if a site’s rules say no, the answer is no from every address. To connect the proxy to a crawler or an agent, see proxies for AI agents.

7.Clean the text

Raw HTML is mostly not content. A model or a retriever that sees menus, cookie notices and footers learns and retrieves noise, and you pay per GB and per token for it.

8.Deduplicate before you index

The same text appears under many URLs: print versions, mirrors, tracking parameters. Duplicates waste storage and tokens, make a retriever return five copies of one answer, and make a model more likely to memorise. Remove exact copies with a hash of the normalised text:

import hashlib
import re
import unicodedata

def normalize(text: str) -> str:
    text = unicodedata.normalize("NFC", text).lower()
    return re.sub(r"\s+", " ", text).strip()

def content_hash(text: str) -> str:
    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()

seen: set[str] = set()

def is_new(text: str) -> bool:
    h = content_hash(text)
    if h in seen:
        return False
    seen.add(h)
    return True

Near copies, such as a page with one changed paragraph, need a similarity measure. Compare sets of word shingles with the Jaccard index:

def shingles(text: str, n: int = 5) -> set[str]:
    words = normalize(text).split()
    return {" ".join(words[i:i + n]) for i in range(max(1, len(words) - n + 1))}

def jaccard(a: set[str], b: set[str]) -> float:
    union = a | b
    return len(a & b) / len(union) if union else 1.0

# pairs above about 0.8 are usually the same article

Comparing every pair does not scale. For a large corpus use MinHash with locality-sensitive hashing, which finds candidate pairs without a full comparison.

9.Record provenance for every document

Provenance is the record of where a document came from, when and how it was collected. Without it you cannot cite a source in a RAG answer, filter by licence or delete a document when its owner asks. Write it next to the text, in the same line of a JSONL file:

import datetime
import json

def make_record(url: str, status: int, text: str, licence: str, country: str) -> dict:
    return {
        "url": url,
        "fetched_at": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
        "status": status,
        "sha256": content_hash(text),
        "licence": licence,          # for example "CC BY 4.0", "terms forbid AI use", "unknown"
        "country": country,          # the vantage point of the fetch
        "text": text,
    }

with open("corpus.jsonl", "a", encoding="utf-8") as f:
    f.write(json.dumps(make_record("https://example.com/docs/a", 200, "Page text", "CC BY 4.0", "UA"),
                       ensure_ascii=False) + "\n")

In a RAG index, store the source URL and the fetch date with every chunk, so the answer can link to its source. Keep documents with the licence value unknown out of any training set until someone has checked them.

10.Store, keep and delete

• Store what you need. Text and metadata, not raw HTML, images and videos, unless the project needs them.
• Set a retention period. Re-fetch or drop old documents on a schedule, and write the schedule down.
• Publish a contact. The page behind your User-Agent should say who runs the crawler and how to reach you.
• Delete on request. Find the documents by domain, URL or hash, remove them from the store and the index, note the deletion, and exclude the source from future training runs.
• Protect the store. Restrict access, encrypt it at rest and keep personal data out of your logs. Our Privacy Policy describes what we keep about your account and traffic.

11.Common mistakes

12.FAQs

Is web scraping for AI training data legal?

It depends on the source, the country and the use. Copyright, terms of use, opt-outs and data protection law all matter, and the rules are still developing. Collect from sources that give you a licence where you can, record your decisions and ask a lawyer before you build a large dataset.

Does robots.txt apply to AI crawlers?

Yes in practice. It is how site owners say what is open, and many name AI crawlers in it. It is a convention and not a law on its own, but ignoring it weakens your position, and in the EU it can serve as a machine-readable reservation of rights.

What is the difference between data for RAG and for training?

RAG keeps copies in an index and cites sources when it answers. Training turns text into model weights and is hard to undo. See the table in section 1.

How fast should an AI crawler go?

Slower than you think. Use one request at a time per site, pause between requests, honor Retry-After and back off on 429 and 5xx answers. If the site publishes a crawl delay, follow it.

Do I need a proxy to collect public data?

Not for every source. Use one when the page depends on the country or when you want one address for the length of a crawl job. See web scraping proxies and proxies for AI and LLM tools.

What should I store with every document?

The URL, the fetch time, the status, a content hash, a licence note, the country of the fetch and the text. That is enough to cite, filter and delete.

Can I use content from a site that forbids AI use?

No. Respect the opt-out and look for a licensed source instead. You can also ask the owner for permission. Once your data pipeline works, check how AI search answers present your own content in each country.

Still have questions about mobile proxies? Visit our FAQ page for detailed answers, or contact our support team — we're here to help.