Use case

Proxies for web scraping

You wrote the scraper on a Sunday. It ran beautifully for 300 pages, and then every response turned into a 403 or a CAPTCHA. That is the site noticing one IP asking for a lot. A proxy spreads the asking around. It will not fix a bad parser or a rude crawl rate.
Sound familiar?

What goes wrong

Sites rarely block a scraper for scraping. They block an address that behaves unlike a person: too many requests, too fast, from a range that belongs to a hosting company.

  • 403s or CAPTCHAs after a few hundred requests
  • Your home IP gets rate-limited and the site is now slow for you in the browser too
  • It worked from your laptop and fails from a cheap VPS, because the VPS range is flagged
  • 429 Too Many Requests, even with a sleep between calls
How a proxy helps

What changes with one

Many addresses where you had one

Rotating residential gives each new connection a different home IP, so the per-IP counter on the other end never climbs far.

Your own IP stays clean

The target sees our exit address. If something gets blocked, it is that address, and your home connection keeps working.

A cheap lane for sites that do not care

Plenty of sites never check whether you come from a hosting range. For those, one rented datacenter IP does the job for a fixed price per term.

When it will not help

A proxy does nothing for a scraper that fires fifty parallel requests at one site, sends a Python user agent, or throws cookies away. Fix those first. A new IP buys a badly behaved crawler a few more minutes.

How not to get blocked →

Which line

What to run it on

Start here

Residential

Sites that bother blocking scrapers usually block by range and by rate. Rotation handles both, and you pay only for the bytes you move.

from $1.75/GB

Or

Datacenter

If a quick test shows the site lets datacenter through, a rented IP for the month is the cheaper way to fetch the same pages.

from $1.50/IP/mo

Back of the envelope

What it would cost

Worked out from today’s price list with the assumptions shown. Your pages will weigh something else, so measure fifty of them and redo the sum.

Residential, by the GB

A one-off crawl of product pages from a single shop

Requests10,000

Weight each~120 KB

Traffic≈ 1.2 GB

Top-up that covers it2 GB

Rate at that size$5.20/GB

Cost$10.40

Assumes HTML only, with images, scripts and fonts not fetched; request and response headers counted. We meter request bytes plus response bytes. The rate is set by the size of one top-up and applies to all of it, and the balance does not expire.

In code

The short version

Swap in your own credentials from the dashboard. It also lists the host and port for each order; where they differ from this example, the dashboard is right.

Full Scrapy setup →

middlewares.py
class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta.setdefault("proxy", "http://USER:[email protected]:8000")

What is off limits

  • Credential stuffing: trying leaked usernames and passwords against a login form
  • Crawling a small site hard enough to slow it down for its real visitors
  • Reselling access to your proxies without a written agreement with us
  • Scraping behind a login you legitimately hold, or high-volume crawling of a small site, are grey areas. Ask in Discord before you start.

From our acceptable use policy. We close accounts that break it, and refund the unused balance when we do.

Web scraping questions

Is web scraping legal?

Collecting publicly available data is lawful in many places, but it depends on where you are, what you collect and the site's terms. Our acceptable use policy allows scraping public data at a considerate rate.

Personal data and copyrighted content carry their own rules, and a proxy does not change them.

Residential or datacenter for my first scraper?

Try datacenter first. If the target returns 403s through datacenter and 200s through residential, the site blocks hosting ranges and you need residential. The residential vs datacenter guide has a script that tells you in about five minutes.

How many GB will my scraper use?

Fetch fifty representative pages, read the byte counts in your usage log, and multiply out. We bill request bytes plus response bytes, headers included, so the log will read a little higher than the size of the HTML you saved.

Do I need a new IP for every request?

For one-off page fetches, rotation per request is the default and usually what you want. Paginating a search or holding a cart needs one address for a while; residential supports sticky sessions for that.

Can I pick which country the IP is in?

Residential has no country selector here. The pool covers 180+ countries and you get whichever one the rotation hands you. ISP and datacenter IPs are bought in the country you choose at checkout.

The community layer

Stuck on web scraping?

Post the target and the error in Discord. Somebody has usually hit the same wall and will tell you whether a proxy is even the fix.

Join the Discord

4,200+monkeys in the Discord

  • Help from humans

    Post your error, get an answer. Usually in minutes, usually from someone who has hit the same wall.

  • A status bot that tells on us

    Pool health, incidents and maintenance posted automatically. Including the bad days.

  • Deals and free traffic

    Bonus GB drops, early access to new pools, and the occasional giveaway for a good bug report.

Join the Discord4,200+ monkeys, free to lurk