Not getting blocked: the boring checklist that works

When a scrape starts failing, the instinct is to buy better proxies. It is the most expensive fix and usually the wrong one. Most blocks are caused by the client, not the IP, and the fixes are free.

Here is the order we would work through. Do not skip ahead — each step is cheaper than the one after it.

1. Read the actual response

Before changing anything, look at what you are being given. The status code and body tell you which problem you have, and they are different problems with different fixes.

• 403 immediately, every request. You are being filtered before you get a chance. Headers or ASN.
• 429, or 200s that degrade over time. Rate limiting. Pacing problem, not an IP problem.
• 200 with a challenge page in the body. Bot detection fired. TLS fingerprint or JavaScript execution.
• Timeouts. Usually the proxy or the target being slow, not a block. Raise the timeout before concluding anything.

Log the body of the first failure. People spend days on the wrong problem because they only logged the status code.

2. Fix your headers

This is free and it fixes more cases than anything else on this list. The default user agent of most HTTP libraries names the library. Sites drop those without thinking about it.

Send a complete, coherent set:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
  (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,\
  image/webp,*/*;q=0.8
Accept-Language: en-GB,en;q=0.9
Accept-Encoding: gzip, deflate, br
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Upgrade-Insecure-Requests: 1

Two things people get wrong. First, header order is itself a fingerprint — real browsers send them in a consistent order and some libraries alphabetise. Second, a rotating user agent with everything else static is worse than a fixed one, because a Chrome-on-Windows string arriving with a macOS Sec-CH-UA-Platform is a louder signal than never rotating at all.

3. Slow down and vary

Perfectly even request spacing does not occur in nature. A request every 1000 ms exactly is a stronger bot signal than a request every 200 ms with human-looking variance.

• Sleep a random interval between requests, not a fixed one
• Back off exponentially on 429, with jitter
• Do not run 200 threads at one hostname because your library allows it
• Spread a long job across hours rather than minutes where you can

If your success rate recovers when you halve the concurrency, you had a pacing problem and no proxy purchase would have fixed it.

4. Handle cookies and sessions properly

Many sites set a cookie on first contact and expect it back. A client that discards cookies looks like a thousand first-time visitors from one address, which is exactly the pattern detection is looking for.

Use a session object that persists cookies, and keep one session tied to one IP. Mixing them — the same cookie appearing from five countries in a minute — is worse than having no cookies at all. If you need both, use sticky sessions so an IP stays put for the life of the cookie.

5. Check your TLS fingerprint

This is where most people stop understanding what is happening. Beyond headers, servers can fingerprint the TLS handshake itself — cipher suites, extensions, the order of both. Python’s requests has a distinctive signature that does not match any browser, and no amount of header spoofing hides it.

Symptom: a clean 200 with a challenge page in the body, while the exact same request from curl in your terminal works. Fix: use a client that mimics a browser handshake, such as curl_cffi in Python, or drive a real browser with Playwright and accept the extra cost.

This is worth checking before you buy anything more expensive. A perfect IP with a Python TLS fingerprint still gets caught.

6. Only now, change proxies

If headers, pacing, cookies and TLS are all sorted and you are still blocked, the IP genuinely is the problem. Move up one step:

• Datacenter → ISP if the block is ASN-shaped but your session needs to stay on one address
• Datacenter → residential if you need many different addresses rather than one trusted one
• Consider country targeting — some sites treat their home country more generously

The residential vs datacenter guide has a five-minute test for which one a target needs, so you are not guessing.

The thing nobody wants to hear

Some targets are genuinely hard, and no proxy makes them easy. Sites with a serious anti-bot vendor, device attestation and behavioural analysis are not solved by buying $2.50/GB traffic. If you have worked through this list and you are still stuck, the honest answer is that you need a real browser, real session warming, and considerably more patience — or a different approach to getting the data.

We would rather tell you that than sell you the most expensive gigabytes we have and watch you burn through them in an afternoon. Ask in Discord before you spend — somebody has probably already fought the target you are looking at.