Go and Cloudflare Turnstile

Proving it from the dependency graph, not the README

Colly's README does not mention JavaScript, browsers or rendering at all. That absence is suggestive rather than conclusive, so the better evidence is its go.mod and its imports.

The require list contains no JavaScript engine — no goja, no otto, no v8go — and no browser driver. Its HTML-related dependencies are goquery for CSS selection, htmlquery and xmlquery for XPath, chardet for encoding detection, plus robotstxt and a URL parser. The main file's imports are standard library networking plus those parsers.

So the pipeline is: net/http fetches bytes, a parser walks them, your callbacks fire. There is no stage at which a script could run, which means there is no flag, option or extension that changes the outcome for a Turnstile page.

Cloudflare describes the same category from their side, listing as unsupported "Command-line tools such as wget, curl, or others that lack JavaScript execution capabilities required for Cloudflare Challenges." The language is not Go-specific; the property is having no JS engine.

What Colly is genuinely excellent at is everything either side of that: over a thousand requests per second on a single core, automatic cookie and session handling, per-domain rate limiting and concurrency, caching, robots.txt support and distributed scraping.

c := colly.NewCollector(
	colly.UserAgent("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"),
)

c.OnRequest(func(r *colly.Request) {
	r.Headers.Set("X-Requested-With", "XMLHttpRequest")
	r.Headers.Set("Referer", "https://www.example.com/")
})

The three options, and their order

First, look for the data behind the page. A challenge protects a document; it does not always protect the API that document calls. This is the cheapest thing to check and it is frequently the answer — open the network panel, filter to XHR, and see whether the JSON you want answers an ordinary request. Colly is then exactly the right tool.

Second, add a browser. Go has two mature CDP drivers, and they have different strengths:

rod's iframe and shadow DOM handling is the one that matters for a Turnstile widget, for the reason the Patchright page goes into: the checkbox sits in an iframe inside a shadow root, and reaching it is a precondition for anything else. Note that rod's last tagged release is considerably older than chromedp's, though its default branch is well ahead of the tag.

// chromedp
ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()

var title string
if err := chromedp.Run(ctx,
	chromedp.Navigate(ts.URL),
	chromedp.Title(&title),
); err != nil {
	log.Fatal(err)
}

Neither contains Turnstile-specific handling, and neither is a stealth-patched driver in the sense that Patchright or nodriver are in Python — they are plain CDP drivers. Cloudflare's statement that automated browsers are not supported for solving production challenges applies to them as much as to anything else.

Third, get the token elsewhere and keep Colly. This preserves what you chose Go for. Colly stays a fast concurrent HTTP client; only the token comes from somewhere with a browser.

Keeping Colly and attaching a token

The OnRequest callback is the natural place, for the same reason a downloader middleware is in Scrapy: it runs before every request and can attach whatever the target needs.

package main

import (
	"bytes"
	"encoding/json"
	"log"
	"net/http"
	"os"

	"github.com/gocolly/colly/v2"
)

type solveReq struct {
	Gate    string `json:"gate"`
	Sitekey string `json:"sitekey"`
	URL     string `json:"url"`
}

type solveRes struct {
	Token  string `json:"token"`
	Status string `json:"status"`
}

func token(sitekey, page string) (string, error) {
	body, _ := json.Marshal(solveReq{Gate: "turnstile", Sitekey: sitekey, URL: page})
	req, err := http.NewRequest("POST", "https://api.solvegate.io/v1/solve", bytes.NewReader(body))
	if err != nil {
		return "", err
	}
	req.Header.Set("Authorization", "Bearer "+os.Getenv("SOLVEGATE_KEY"))
	req.Header.Set("Content-Type", "application/json")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return "", err
	}
	defer res.Body.Close()

	var out solveRes
	if err := json.NewDecoder(res.Body).Decode(&out); err != nil {
		return "", err
	}
	return out.Token, nil
}

func main() {
	c := colly.NewCollector()

	t, err := token("0x4AAAAAAA_target", "https://app.example.com/login")
	if err != nil {
		log.Fatal(err)
	}

	// Post the token as the field the widget would have filled.
	if err := c.Post("https://app.example.com/login", map[string]string{
		"username":              "someone@example.com",
		"password":              os.Getenv("APP_PASSWORD"),
		"cf-turnstile-response": t,
	}); err != nil {
		log.Fatal(err)
	}

	c.OnResponse(func(r *colly.Response) {
		log.Println("response received", r.StatusCode)
	})

	c.Visit("https://app.example.com/")
}

Two details decide whether this works. The field name is cf-turnstile-response unless the page renamed it with data-response-field-name — our checker reports the real one along with the sitekey. And Colly handles cookies automatically through its own jar, which is what you want when a clearance cookie is involved.

Import path note: use github.com/gocolly/colly/v2. The unversioned path is stuck on a 2019 release.

Costs, honestly

The reason to keep Colly is throughput. A browser per worker is orders of magnitude more memory and far fewer concurrent pages, and that is usually the thing Go was chosen to avoid.

Compare total API request time against launching a browser on the same test property. Queue wait, proxy behavior and retries affect that comparison. Prepaid credits run from $0.40 per 1,000 solves down to $0.075 at volume; failed solves are never billed; the first 1,000 are free.

And if the site is yours and the challenge is only in the way of a test, none of this is the right answer — Cloudflare's dummy sitekeys remove it from the test entirely, which is what they exist for. Solve only against properties you own or are authorised to test.

Common questions

Related

Sources

• Colly — README, go.mod and examples
• chromedp
• rod
• Cloudflare — unsupported environments

More in Guides