Your first scraper with Python and rotating proxies

This is a working scraper in about forty lines of Python. It rotates proxies, retries politely, and counts its own bytes so you can see what you are spending while you learn. No frameworks, no scaffolding, nothing to install beyond requests.

What you need first

• Python 3.9 or newer
• pip install requests
• Proxy credentials — a host, a port, a username and a password

This guide uses the rotating residential endpoint, because rotation is what the title promised and residential is the line billed by the byte, which the counter at the end measures. If your target does not block hosting ranges, a datacenter IP is the cheaper way to run volume, and switching means putting your address’s IP and port where the gateway is: http://USER:PASS@IP:PORT. The residential vs datacenter test tells you which one a target needs, and current rates are on the pricing page.

Step 1: one request through a proxy

A proxy in requests is a dictionary of scheme to URL. The credentials go in the URL, which is ugly but standard.

import requests

PROXY = "http://USER:[email protected]:8000"
proxies = {"http": PROXY, "https": PROXY}

r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=20)
print(r.json())

Run it twice. The IP should be different each time, because the residential endpoint rotates on every request. On a datacenter address it will be the same both times, since that address is the one you rented.

Step 2: look like a browser

The default requests user agent announces itself as python-requests, and plenty of sites drop it on sight. This is the single highest-value change you can make and it is free.

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
}

Send a realistic set, not just a user agent. A request claiming to be Chrome that sends no Accept-Language is a tell.

Step 3: the scraper

Here is the whole thing. It reads URLs, fetches each one with retries and backoff, records the outcome, and totals the bytes it moved.

import csv, random, sys, time
import requests

PROXY = "http://USER:[email protected]:8000"
PROXIES = {"http": PROXY, "https": PROXY}
MAX_RETRIES = 3
bytes_moved = 0


def fetch(url, session):
    """Return (status, text) or (None, None). Retries with backoff."""
    global bytes_moved

    for attempt in range(MAX_RETRIES):
        try:
            r = session.get(url, proxies=PROXIES, timeout=25)
        except requests.RequestException as exc:
            print(f"  network error: {exc}", file=sys.stderr)
        else:
            bytes_moved += (
                len(str(r.request.headers))
                + sum(len(k) + len(v) for k, v in r.headers.items())
                + len(r.content)
            )

            if r.status_code == 200:
                return r.status_code, r.text
            if r.status_code in (429, 503):
                print(f"  rate limited ({r.status_code})", file=sys.stderr)
            else:
                # 403, 404 and friends will not improve by asking again.
                return r.status_code, None

        # Exponential backoff with jitter, so retries do not sync up.
        time.sleep(2 ** attempt + random.random())

    return None, None


def main(path):
    session = requests.Session()
    session.headers.update(HEADERS)

    with open(path) as f, open("results.csv", "w", newline="") as out:
        writer = csv.writer(out)
        writer.writerow(["url", "status", "length"])

        for line in f:
            url = line.strip()
            if not url:
                continue

            status, body = fetch(url, session)
            writer.writerow([url, status, len(body) if body else 0])
            print(f"{status}  {url}")

            # Be a considerate guest.
            time.sleep(random.uniform(0.5, 1.5))

    # Your rate per GB. This is our single-gigabyte rung when this page was built.
    rate_per_gb = 2.40

    mb = bytes_moved / 1_048_576
    print(f"\nmoved {mb:.2f} MB  = ${mb / 1024 * rate_per_gb:.4f}")


if __name__ == "__main__":
    main(sys.argv[1])

Run it with a file of URLs, one per line:

python scraper.py urls.txt

The parts that matter

Retry only what is worth retrying

A 429 or a 503 is temporary and worth another attempt. A 403 or a 404 will not change because you asked twice, and retrying them just spends money. The code above distinguishes the two, which is the difference between a scraper and a denial-of-service tool.

Backoff with jitter

2 ** attempt + random.random() waits roughly one, two and four seconds. The random component matters when you eventually run this concurrently: without it, every worker that fails at the same moment retries at the same moment.

Count your own bytes

The bytes_moved counter is the important habit. We bill request bytes plus response bytes, so it counts both sides. It is a rough number: it misses TLS overhead, and r.content is the body after decompression, which is larger than what crossed the wire for a gzipped page. It still gives you a reference to compare against your provider’s meter. The rate in the script is $2.40/GB, our single-gigabyte price when this page was built; bigger orders are cheaper per GB. If their figure is dramatically higher than yours, that is worth investigating; see how providers inflate usage.

Sleep between requests

The half-to-one-and-a-half second pause is not politeness theatre. Steady machine-gun pacing is one of the easiest bot signals to detect, and sites that would have served you happily at a human pace will start refusing.

When it stops working

It eventually will, and the order of things to try is: fix your headers, slow down, rotate more, and only then buy more expensive proxies. The blocking checklist goes through that in order, and the error field guide explains what each status code is telling you. Buying pricier proxies should be the fourth thing you try.

If it works but runs too slowly, the next step is running requests concurrently rather than one after another. Async scraping with httpx and aiohttp takes this scraper there without hammering the target.

For curl, Node and other scrapers, see Plugging the proxy into curl, Python, Node and scrapers.

Stuck? Paste the error in our Discord. Somebody has hit it.