Your first scraper with Python and rotating proxies

This is a working scraper in about forty lines of Python. It rotates proxies, retries politely, and counts its own bytes so you can see what you are spending while you learn. No frameworks, no scaffolding, nothing to install beyond requests.

What you need first

• Python 3.9 or newer
• pip install requests
• Proxy credentials — a host, a port, a username and a password

Start on datacenter proxies. At $0.45/GB, everything in this guide costs you a few cents, and you can move to residential later by changing one hostname.

Step 1: one request through a proxy

A proxy in requests is a dictionary of scheme to URL. The credentials go in the URL, which is ugly but standard.

import requests

PROXY = "http://USER:[email protected]:8000"
proxies = {"http": PROXY, "https": PROXY}

r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=20)
print(r.json())

Run it twice. The IP should be different each time, because rotation happens per request by default. If it is not different, you have a sticky session configured somewhere.

Step 2: look like a browser

The default requests user agent announces itself as python-requests, and plenty of sites drop it on sight. This is the single highest-value change you can make and it is free.

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
}

Send a realistic set, not just a user agent. A request claiming to be Chrome that sends no Accept-Language is a tell.

Step 3: the scraper

Here is the whole thing. It reads URLs, fetches each one with retries and backoff, records the outcome, and totals the bytes it moved.

import csv, random, sys, time
import requests

PROXY = "http://USER:[email protected]:8000"
PROXIES = {"http": PROXY, "https": PROXY}
MAX_RETRIES = 3
bytes_moved = 0


def fetch(url, session):
    """Return (status, text) or (None, None). Retries with backoff."""
    global bytes_moved

    for attempt in range(MAX_RETRIES):
        try:
            r = session.get(url, proxies=PROXIES, timeout=25)
        except requests.RequestException as exc:
            print(f"  network error: {exc}", file=sys.stderr)
        else:
            bytes_moved += len(r.content) + len(str(r.request.headers))

            if r.status_code == 200:
                return r.status_code, r.text
            if r.status_code in (429, 503):
                print(f"  rate limited ({r.status_code})", file=sys.stderr)
            else:
                # 403, 404 and friends will not improve by asking again.
                return r.status_code, None

        # Exponential backoff with jitter, so retries do not sync up.
        time.sleep(2 ** attempt + random.random())

    return None, None


def main(path):
    session = requests.Session()
    session.headers.update(HEADERS)

    with open(path) as f, open("results.csv", "w", newline="") as out:
        writer = csv.writer(out)
        writer.writerow(["url", "status", "length"])

        for line in f:
            url = line.strip()
            if not url:
                continue

            status, body = fetch(url, session)
            writer.writerow([url, status, len(body) if body else 0])
            print(f"{status}  {url}")

            # Be a considerate guest.
            time.sleep(random.uniform(0.5, 1.5))

    mb = bytes_moved / 1_048_576
    print(f"\nmoved {mb:.2f} MB  = ${mb / 1024 * 0.45:.4f} at $0.45/GB")


if __name__ == "__main__":
    main(sys.argv[1])

Run it with a file of URLs, one per line:

python scraper.py urls.txt

The parts that matter

Retry only what is worth retrying

A 429 or a 503 is temporary and worth another attempt. A 403 or a 404 will not change because you asked twice, and retrying them just spends money. The code above distinguishes the two, which is the difference between a scraper and a denial-of-service tool.

Backoff with jitter

2 ** attempt + random.random() waits roughly one, two and four seconds. The random component matters when you eventually run this concurrently: without it, every worker that fails at the same moment retries at the same moment.

Count your own bytes

The bytes_moved counter is the important habit. It is a rough number — it misses TLS overhead — but it gives you a reference to compare against your provider’s meter. If their figure is dramatically higher than yours, that is worth investigating; see how providers inflate usage.

Sleep between requests

The half-to-one-and-a-half second pause is not politeness theatre. Steady machine-gun pacing is one of the easiest bot signals to detect, and sites that would have served you happily at a human pace will start refusing.

When it stops working

It eventually will, and the order of things to try is: fix your headers, slow down, rotate more, and only then buy more expensive proxies. The blocking checklist goes through that in order. Upgrading to residential should be the fourth thing you try, not the first.

Stuck? Paste the error in our Discord. Somebody has hit it.