How proxy providers inflate your usage, and how to catch them

If you have ever finished a scraping job, looked at your provider’s usage counter, and thought that is more than I downloaded, you were probably right. Usage inflation is the most common way a proxy provider makes a low advertised rate profitable, and it works because almost nobody checks.

This post is about the three ways it happens, and a test you can run in an afternoon that tells you whether it is happening to you.

Why the meter is where the money is

A proxy provider has exactly one number that turns traffic into revenue: the byte count. If you buy 100 GB at $1.50 and the counter runs 25% hot, the provider has quietly charged you $1.88/GB and you have no way to tell from the invoice. It looks identical to a low rate honestly delivered.

Compare that with the alternatives. Raising the advertised price is visible on a comparison page. Degrading the pool is visible in your success rate. Inflating the meter is invisible unless the customer runs their own counter, which almost none do. It is the cheapest lie available, so it is the common one.

The three ways it happens

1. Counting things that never reached you

The honest definition of billable traffic is: bytes that crossed the provider’s network on your behalf. That means your request headers and body, plus the response headers and body, plus the TLS overhead on the tunnel.

The dishonest version adds connection attempts that failed before carrying data, DNS lookups, internal health checks on the exit node, and — the popular one — retries that the provider performed automatically without telling you. A provider that silently retries a failed request three times and bills all four attempts has tripled your bill on the worst-performing part of your job.

Ask any provider one question: “Do you bill for connections that failed before transferring data, and do you bill for retries you performed on my behalf?” A provider with a clean answer will give you one immediately.

2. Rounding per request instead of per account

This one is subtle and very effective. If usage is rounded up to the nearest kilobyte per request, then a job making a million small API calls of 300 bytes each gets billed for a million kilobytes instead of 300 megabytes — more than triple.

Rounding should happen once, at the bottom of your invoice, not a million times on the way there. If a provider will not tell you their rounding granularity, assume it is per request.

3. Just multiplying

The least sophisticated version, and it exists. The counter is simply the real number times some constant slightly above one. It survives because the customer has no reference measurement, and because a 15% discrepancy is inside the range that a reasonable person will attribute to “protocol overhead I do not fully understand.”

The test

You need two numbers for the same window of traffic: what you actually moved, and what the provider says you moved. Getting the first one is the only part that takes effort.

Step 1: build a fixed job

Pick a few hundred URLs and fetch them. Do not use a live production crawl — you want something you can run twice and compare. Turn off any retry logic in your own client so the request count is deterministic.

# urls.txt has 500 lines
while read url; do
  curl -s -o /dev/null \
    -w '%{size_download} %{size_upload} %{size_header} %{size_request}\n' \
    -x http://USER:[email protected]:8000 \
    "$url"
done < urls.txt | awk '{s+=$1+$2+$3+$4} END {print s " bytes"}'

That sums the four size fields curl reports: response body, request body, response headers and request headers. It undercounts slightly because it does not see TLS record overhead, which is the point — you expect the provider’s number to be a little higher than yours.

Step 2: get the provider’s number

Export usage for the same window. If your provider only shows a daily total with no per-request breakdown, run the test on an otherwise idle day so the daily total is your test. If they cannot give you a number for a specific window at all, you have learned something already.

Step 3: read the gap

• 0–8% over your count. Normal. That is TLS overhead and connection setup that curl did not report.
• 8–15%. Worth a question. Ask what is included. A good answer exists; make them give it.
• Above 20%. Something is being counted that you did not ask for. Retries and failed connections are the usual suspects.
• Under your count. Rare, and usually means they bill response body only. Enjoy it.

Run it twice on different days before you accuse anyone of anything. A single measurement can be thrown off by a target that redirected more than usual.

What to do with the answer

If the gap is large, ask the provider to explain it before you leave. Sometimes the explanation is legitimate and boring — you were hitting a lot of redirects, or your client was retrying without you realising. A provider that engages with the numbers is worth keeping. A provider that responds with “overhead” and no detail is telling you what you needed to know.

Doing this to us

Everything above applies to ProxyMonkey and we would genuinely rather you ran it on day one with $5 than on day ninety with a real bill. Our usage export is one row per request with bytes in, bytes out and cost, so step two is a CSV download rather than an argument.

If our number comes out higher than yours by more than protocol overhead explains, that is a bug and we refund the difference. The honesty page goes through exactly what we count and what we do not.