Scrapy and Cloudflare Turnstile
Scrapy is a scheduler and an HTTP client. There is no browser in its architecture and no JavaScript engine, so a page carrying Turnstile comes back as markup with the data missing. That leaves three routes, and Scrapy's own documentation is unusually clear about which to try first — it is not the one most tutorials reach for.
Why the response is empty
Scrapy's architecture has an engine, a scheduler, a downloader, spiders, item pipelines and two middleware layers. It does not have a rendering component, and nothing in it executes page scripts. The downloader fetches bytes; selectors run over those bytes.
A Turnstile page's useful content is behind a challenge that only exists as JavaScript: api.js runs, probes the environment, negotiates inside an iframe, and writes a token into a hidden cf-turnstile-response input. None of that happens in a process with no JS engine, so the selector that works in your browser's inspector matches nothing in Scrapy.
Cloudflare names the category from their side, listing as unsupported "Command-line tools such as wget, curl, or others that lack JavaScript execution capabilities required for Cloudflare Challenges." A pure HTTP client is in that group whatever language it is written in.
What Scrapy's own docs tell you to do first
The *Selecting dynamically-loaded content* page opens with advice that is worth following before anything else:
"Some webpages show the desired data when you load them in a web browser. However, when you download them using Scrapy, you cannot reach the desired data using selectors. When this happens, the recommended approach is to find the data source and extract the data from it. If you fail to do that, and you can nonetheless access the desired data through the DOM from your web browser, see Using a headless browser."
Find the data source first. Adding a browser is explicitly the fallback, and the same page explains why: "reproducing those requests that contain the desired data is the preferred approach. The effort is often worth the result: structured, complete data with minimum parsing time and network transfer."
This is worth taking seriously on a Turnstile page specifically, because the challenge protects a document, not necessarily an API. It is common for the gated page to be the HTML while the underlying JSON endpoint is reachable with an ordinary request. Open the network panel, sort by XHR, and look before you install anything.
One distinction that trips people: the same docs page has a section on *parsing* JavaScript, using regular expressions, chompjs or js2xml. That is about extracting data embedded in script source text. It never executes anything, and it will not produce a Turnstile token.
Option two: add a browser
If the data genuinely only exists after scripts run, Scrapy's documentation recommends scrapy-playwright over driving Playwright directly, and gives the reason: "using playwright-python directly as in the above example circumvents most of the Scrapy components (middlewares, dupefilter, etc). We recommend using scrapy-playwright for a better integration."
# settings.py DOWNLOAD_HANDLERS = { "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", # "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", } TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
It is opt-in per request, which is the property that makes it usable — the handler inherits from the default one, so only requests you mark actually launch a browser:
import scrapy class AwesomeSpider(scrapy.Spider): name = "awesome" async def start(self): # start_requests in Scrapy < 2.13 yield scrapy.Request("https://httpbin.org/get", meta={"playwright": True}) def parse(self, response, **kwargs): # response contains the page as seen by the browser return {"url": response.url}
The alternative, scrapy-splash, needs a separate Splash server running alongside your crawl — its README says so plainly — which is a second moving part to deploy and monitor.
Be clear-eyed about what a browser buys you here. It gets the DOM after scripts run. It does not make an automated client supported: Cloudflare states that "Browser automation frameworks, such as Selenium, Puppeteer, Playwright, and Cypress, are not supported for solving production challenges." And a browser inside every Scrapy worker costs memory and concurrency, which is most of what you chose Scrapy for.
Option three: keep Scrapy, get the token elsewhere
The third route keeps the architecture intact. Scrapy stays a fast HTTP client and scheduler; only the token comes from somewhere with a browser. A downloader middleware is the natural home, because process_request() runs before every request and can attach whatever the target needs.
# settings.py — lower numbers sit closer to the engine DOWNLOADER_MIDDLEWARES = { "myproject.middlewares.TurnstileMiddleware": 543, }
# myproject/middlewares.py import json import os import urllib.request SITEKEY = "0x4AAAAAAA_target" class TurnstileMiddleware: """Attach a fresh Turnstile token to requests that ask for one. Opt in per request with meta={"turnstile": True}. Returning None lets the request continue down the middleware chain, which is what the documented process_request() contract expects. """ def process_request(self, request, spider): if not request.meta.get("turnstile"): return None body = json.dumps({ "gate": "turnstile", "sitekey": SITEKEY, "url": request.url, }).encode() req = urllib.request.Request( "https://api.solvegate.io/v1/solve", data=body, headers={ "Authorization": f"Bearer {os.environ['SOLVEGATE_KEY']}", "Content-Type": "application/json", }, ) with urllib.request.urlopen(req, timeout=30) as res: solve = json.load(res) request.cookies["cf_clearance"] = solve.get("token", "") return None
Two accuracy notes that cost people afternoons. Set cookies through the cookies argument, never a Cookie header — Scrapy's own documentation carries the caution: "Cookies set via the Cookie header are not considered by the cookie middleware. If you need to set cookies for a request, use the cookies argument." And process_request() must return None, a Response, or a Request, or raise IgnoreRequest; returning anything else is a bug that will not announce itself clearly.
For a form submission rather than a cookie, the token goes in as an ordinary field:
from scrapy import FormRequest yield FormRequest( url="https://app.example.com/login", formdata={ "username": "...", "password": "...", "cf-turnstile-response": token, }, callback=self.after_login, )
Do not reach for FormRequest.from_response(). As of Scrapy 2.18 it emits a ScrapyDeprecationWarning pointing at the form2request library, and the documentation now says to use that instead. Plain FormRequest(url=..., formdata=...) is not deprecated. Most tutorials on this subject predate the change.
Choosing between them
| Route | Good when | What it costs |
|---|---|---|
| Find the underlying data source | Almost always try first — Scrapy's own recommendation | Some time in the network panel, and it sometimes is not there |
| scrapy-playwright | The data genuinely only exists post-render, or you need a screenshot | A browser per worker; automation is still unsupported for production challenges |
| Token from a solver, Scrapy unchanged | The gate is the only obstacle and throughput matters | Per-solve cost; the target must be yours or authorised |
If the target is your own site and the challenge is only in the way of a test, none of these is the right answer — Cloudflare's dummy sitekeys remove the challenge from the test entirely, which is what they exist for.
A token typically comes back in under 1.5 seconds, which is the figure worth comparing against a browser launch. Prepaid credits run from $0.40 per 1,000 down to $0.075 at volume; failed solves are never billed; the first 1,000 are free. The solve reference has the whole call.
Common questions
No. Scrapy is a scheduler and an HTTP client with no browser and no JavaScript engine, and a Turnstile token is the output of running JavaScript. Cloudflare lists command-line tools that lack JS execution as unsupported environments for challenges, and a pure HTTP client is in that group.
Scrapy's own docs recommend scrapy-playwright, and specifically recommend it over driving Playwright directly because direct use bypasses most Scrapy components — middlewares, the dupefilter and so on. scrapy-splash needs a separate Splash server running alongside your crawl, which is another moving part to deploy and watch.
Into formdata as cf-turnstile-response for a form submission, or into request.cookies for a clearance cookie. Use the cookies argument rather than a Cookie header — Scrapy's documentation warns that cookies set via the header are not seen by the cookie middleware.
It was deprecated in Scrapy 2.18 in favour of the form2request library, and calling it emits a ScrapyDeprecationWarning. It still works. Plain FormRequest(url=..., formdata=...) is not deprecated and is the right shape for attaching a token.
Often, and Scrapy's docs put it first: find the data source and hit it directly. A challenge frequently protects a rendered document while the JSON endpoint behind it answers an ordinary request. That is faster, less to parse, and less to maintain than either a browser or a solver.
Related
Sources
More in Guides
Automating a gate you own or are authorised to test?
// SolveGate clears Cloudflare Turnstile and WAF challenges through one REST call · first 1,000 solves free