ProxyMonkey
DiscordStart with $5
Start $5
Go · setup guide

Colly (Go) proxy setup

Colly runs on net/http, so a proxy is a URL with the credentials in it, and Go sends them on every CONNECT. SetProxy puts that URL on the collector's transport and switches keep-alives off in the same call. That is why every request rotates on the residential gateway, and why a static IP wants a transport of your own. For plain net/http without a crawler, see the Go guide.
You will need

Residentialresi.proxymonkey.io:8000

ISP or datacenterIP:PORT

CredentialsUSER:PASS

Copy your own from the dashboard, which also lists the host and port for every order. Where it differs from this page, the dashboard is right.

Install

Go 1.24 or newer, inside a module. Colly v2 lives under the /v2 import path.

terminal
go mod init monkeycrawl
go get github.com/gocolly/colly/v2

Rotating residential

SetProxy sets the proxy and DisableKeepAlives on the collector's transport, so every request opens a new tunnel through the residential gateway and leaves from a new address. AllowURLRevisit lets the loop fetch the same URL three times, and the timeout goes from Colly's default 10 seconds to 30, which suits a slow residential exit.

main.go
package main

import (
	"fmt"
	"log"
	"time"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(colly.AllowURLRevisit())
	c.SetRequestTimeout(30 * time.Second)
	if err := c.SetProxy("http://USER:[email protected]:8000"); err != nil {
		log.Fatal(err)
	}

	c.OnResponse(func(r *colly.Response) {
		fmt.Print(string(r.Body))
	})
	c.OnError(func(r *colly.Response, err error) {
		log.Println(r.Request.URL, err)
	})

	for i := 0; i < 3; i++ {
		c.Visit("https://httpbin.org/ip")
	}
}

A static datacenter or ISP IP

On a static datacenter IP you want the opposite: connections that stay open and get reused. SetProxy would switch keep-alives off again, so hand the collector a transport of your own with WithTransport, the proxy on it, and do not call SetProxy afterwards. This one crawls with up to eight requests in flight. For an ISP order, use the host and port your dashboard lists for it.

main.go
package main

import (
	"log"
	"net/http"
	"net/url"
	"time"

	"github.com/gocolly/colly/v2"
)

func main() {
	proxy, err := url.Parse("http://USER:PASS@IP:PORT")
	if err != nil {
		log.Fatal(err)
	}

	c := colly.NewCollector(colly.Async(true))
	c.WithTransport(&http.Transport{
		Proxy:               http.ProxyURL(proxy),
		MaxIdleConnsPerHost: 8,
	})
	c.SetRequestTimeout(30 * time.Second)
	c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 8})

	c.OnResponse(func(r *colly.Response) {
		log.Println(r.StatusCode, r.Request.URL)
	})

	for _, u := range []string{"https://httpbin.org/ip", "https://httpbin.org/headers"} {
		c.Visit(u)
	}
	c.Wait()
}

Keeping one identity

Each collector from NewCollector has its own in-memory cookie jar, shared by every Visit on it. With SetProxy on the rotating gateway, keep-alives are off, so those cookies arrive from a new address on every request, which a site that ties a session to an IP will notice. For one identity per collector, create each with NewCollector and give it a static IP or a residential sticky session; the session setting for your account is in the dashboard. Clone() is not a new identity: a clone shares the original's cookie jar and transport, proxy included.

Specific to Colly (Go)

Things worth knowing

Several static IPs in turn

proxy.RoundRobinProxySwitcher hands each request the next URL in its list. The residential gateway already rotates by itself, so this is for spreading a crawl over several static datacenter or ISP IPs. SetProxyFunc switches keep-alives off just as SetProxy does.

switcher.go
import "github.com/gocolly/colly/v2/proxy"

rp, err := proxy.RoundRobinProxySwitcher(
	"http://USER:PASS@IP_1:PORT_1",
	"http://USER:PASS@IP_2:PORT_2",
)
if err != nil {
	log.Fatal(err)
}
c.SetProxyFunc(rp)

Retry from OnError

Colly has no retry setting. r.Request.Retry() sends a failed request again with the same context, so a counter kept there caps the attempts. On the rotating gateway each retry leaves from a new address, and each one is metered.

retry.go
c.OnError(func(r *colly.Response, err error) {
	tries, _ := r.Ctx.GetAny("tries").(int)
	if tries < 2 {
		r.Ctx.Put("tries", tries+1)
		r.Request.Retry()
	}
})

len(r.Body) is the unzipped size

We meter request bytes plus response bytes, headers included. r.Body is not the wire size either: Go unzips the gzip it asked for, and Colly unzips gzip you asked for, but only gzip, so a body requested as br or deflate stays compressed. Ask for gzip yourself and Content-Length keeps the size that crossed the wire; fall back to len(r.Body) when a server sends none. The sum skips the request and status lines, and TLS overhead makes ours slightly higher.

wire_bytes.go
headerBytes := func(h *http.Header) (n int) {
	for k, vs := range *h {
		for _, v := range vs {
			n += len(k) + len(v) + 4
		}
	}
	return n
}

c.OnRequest(func(r *colly.Request) {
	r.Headers.Set("Accept-Encoding", "gzip")
})

wire := 0
c.OnResponse(func(r *colly.Response) {
	body := len(r.Body)
	if n, err := strconv.Atoi(r.Headers.Get("Content-Length")); err == nil {
		body = n
	}
	wire += headerBytes(r.Request.Headers) + headerBytes(r.Headers) + body
})

c.Visit("https://example.com/")
fmt.Println("bytes:", wire)
When it breaks

Common errors and fixes

  • Get "https://...": Proxy Authentication Required in OnError, or the same text with status 407 on an http:// URL

    Why
    The gateway answered with a 407: the credentials were rejected. On an HTTPS URL the CONNECT failed and there is no response; on plain HTTP Colly turns the 407 response into the error.
    Fix
    Build the URL with url.UserPassword so a password with @ or : is escaped, pass its String() to SetProxy, and copy the pair fresh from the dashboard.

  • proxyconnect tcp: dial tcp: lookup ... on ...:53: no such host from a Visit

    Why
    A typo in the proxy host, or DNS that cannot resolve it from where the crawler runs. connection refused in the same place means the wrong port.
    Fix
    Compare the host and port with the dashboard, and check the error SetProxy returns: invalid port ":PORT" after host means the placeholders are still in the URL.

  • context deadline exceeded (Client.Timeout exceeded while awaiting headers) after ten seconds

    Why
    Colly gives every request 10 seconds by default, short for a slow residential exit.
    Fix
    Raise it with SetRequestTimeout(30 * time.Second) and retry from OnError. With keep-alives off on the rotating gateway, the retry leaves from a new address.

  • tls: failed to verify certificate: x509: certificate signed by unknown authority

    Why
    A scratch or minimal image with no CA certificates, or a company CA the system does not trust. TLS runs through the tunnel to the target; the proxy is not part of it.
    Fix
    Install ca-certificates in the image. For a private CA, set TLSClientConfig on a transport passed to WithTransport before SetProxy, which keeps an existing *http.Transport and only changes its proxy and keep-alives.

A failed connection that moved no data is not billed. Retries you send are billed like any other request, and each shows as its own line in your usage log.

Other setup guides

Start with $5All guides

Colly (Go) is named here only to say what this guide covers. We are not affiliated with its authors.

The community layer

Colly (Go) still misbehaving?

Paste the error and the few lines that set up the proxy into Discord, with the password taken out. Someone there has seen it before.

Join the Discord

4,200+monkeys in the Discord

  • Help from humans

    Post your error, get an answer. Usually in minutes, usually from someone who has hit the same wall.

  • A status bot that tells on us

    Pool health, incidents and maintenance posted automatically. Including the bad days.

  • Deals and free traffic

    Bonus GB drops, early access to new pools, and the occasional giveaway for a good bug report.

Join the Discord4,200+ monkeys, free to lurk