Scrapy Rotating Proxies: A Practical Setup Guide

TL;DR

Use a rotating gateway for the simplest deployment, a local pool for per-proxy control, and custom middleware for route-specific logic. Keep sticky sessions for stateful workflows, rotate retryable public requests to a new route, and verify the exit IP before increasing concurrency. Test all settings against the target site’s access rules.

Choose a proxy architecture

Rotating proxies are not just an IP-switching feature. Stable crawling depends on how requests are routed, how failed routes are retried, and whether stateful requests keep the same session. Pick the operating model before adding concurrency.

Use a gateway for a single provider endpoint. Use a proxy pool when you already buy or curate individual proxies. Use custom middleware when one spider must treat public pages, login flows, and other stateful workflows differently.

A managed alternative: proxy APIs

Some providers expose a scraping or proxy API instead of a conventional proxy endpoint. This is a managed alternative, not a fourth local rotation architecture: the spider sends a target URL and options to the API, while the provider may manage proxy selection, retries, rendering, or geo settings upstream. It can be useful when the team values a managed request workflow over direct control of Scrapy’s downloader path.

It is not a replacement for every proxy use case. API integrations can change request and response formats, add provider-specific limits, and make direct network troubleshooting less visible. Use a gateway or local pool when you need standard HTTP proxy behavior, direct proxy observability, or fine-grained request routing inside Scrapy.

Method 1: Use a rotating proxy gateway

With a gateway, Scrapy connects to one endpoint and the provider handles exit rotation. Scrapy’s HttpProxyMiddleware documentation supports assigning a proxy through Request.meta["proxy"].

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}

HTTPPROXY_ENABLED = True
PROXY_GATEWAY = "http://user:password@gateway.example.com:9000"

# spider.py
import scrapy
from myproject.settings import PROXY_GATEWAY


class ProductSpider(scrapy.Spider):
    name = "product_spider"
    start_urls = ["https://example.com/products"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, callback=self.parse, meta={"proxy": PROXY_GATEWAY})

    def parse(self, response):
        yield {"url": response.url}

This model avoids maintaining exit IPs in the spider. It can also support geo-targeting and sticky sessions when the provider exposes those options. Confirm endpoint syntax, authentication, countries, and session parameters in the provider documentation. For Rola IP, start with the proxy parameter documentation.

The gateway still needs crawl controls. Do not respond to every 403 with unlimited retries, and do not use high concurrency before confirming how often one exit is reused.

Method 2: Use a Scrapy proxy pool

Choose a local pool when you need to select, pause, and replace individual proxy URLs. scrapy-rotating-proxies is a common package for this purpose. As of July 24, 2026, its PyPI page lists version 0.6.2 with an Alpha development status, so test it against your Scrapy version and downloader stack before production use.

pip install scrapy-rotating-proxies

ROTATING_PROXY_LIST = [
    "http://user:pass@1.2.3.4:8000",
    "http://user:pass@5.6.7.8:8000",
]

DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}

For a file-backed list, set ROTATING_PROXY_LIST_PATH = "/path/to/proxies.txt".

The package can track proxy health and make concurrency effectively per proxy for proxied requests. It does not supply proxy quality or site-specific ban detection. Requests with request.meta["proxy"] already set are not managed by the package, which is useful for exceptions but important to document in mixed routing setups.

Treat 403, 429, challenge pages, empty templates, and abnormal redirects as separate observations. A small pool plus high concurrency tends to reuse exits too quickly; increasing retries will usually make that worse.

What the middleware actually manages

The package does more than choose a random URL. RotatingProxyMiddleware records whether a route is alive or dead through the _ban request metadata set by BanDetectionMiddleware. A dead route is not discarded forever: it is checked again after randomized exponential backoff. This prevents a temporarily unhealthy route from being selected for every immediate retry while still allowing recovery.

Its retry boundary matters. ROTATING_PROXY_PAGE_RETRY_TIMES defines how many different proxies may be tried before the package treats the result as a page failure rather than a proxy failure. If this is too high, an incorrectly detected block can consume healthy routes. If it is too low, a transient route failure may be misclassified as a target-page problem.

Default Scrapy concurrency settings become effectively per proxy for proxied requests when this middleware is enabled. Therefore, CONCURRENT_REQUESTS_PER_DOMAIN = 2 can mean up to two concurrent connections through each active proxy, rather than two requests total. Calculate load from the active pool size, not from one global setting alone.

Customize ban detection for the target response

HTTP status alone is often incomplete. A site can return 200 OK with a challenge page or an empty shell. Define the page signals that represent an unusable response for your authorized use case and test them against saved examples.

from rotating_proxies.policy import BanDetectionPolicy


class StoreBanPolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        text = response.text.lower()
        if response.status in {403, 429}:
            return True
        if "verify you are human" in text or "captcha" in text:
            return True
        if response.status == 200 and "product-grid" not in text:
            return True
        return False

ROTATING_PROXY_BAN_POLICY = "myproject.policies.StoreBanPolicy"

Only add a content marker that is stable for the intended page type. A generic string can mark valid pages as banned and unnecessarily retire good routes. For spiders with several page types, use route-aware conditions or separate spiders instead of one broad keyword rule.

Method 3: Build custom proxy middleware

Custom downloader middleware is appropriate when routing depends on the URL, page type, or session. Scrapy lets middleware modify or reschedule a request in process_request, process_response, and process_exception; see the downloader middleware guide.

import random


class CustomProxyMiddleware:
    def __init__(self, proxies, max_retries=3):
        self.proxies = proxies
        self.max_retries = max_retries

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("CUSTOM_PROXY_LIST"),
                   crawler.settings.getint("CUSTOM_PROXY_RETRY_TIMES", 3))

    def pick_proxy(self, excluded=None):
        candidates = [item for item in self.proxies if item != excluded] or self.proxies
        if not candidates:
            raise RuntimeError("CUSTOM_PROXY_LIST is empty")
        return random.choice(candidates)

    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            request.meta["proxy"] = self.pick_proxy()

    def retry_with_new_proxy(self, request, spider):
        retries = request.meta.get("proxy_retry_times", 0) + 1
        if retries > self.max_retries:
            return None
        retry = request.copy()
        retry.dont_filter = True
        retry.meta["proxy_retry_times"] = retries
        retry.meta["proxy"] = self.pick_proxy(request.meta.get("proxy"))
        return retry

    def is_retryable_response(self, response):
        if response.status == 407:
            return False  # Fix proxy credentials or allowlisting before retrying.
        return response.status in {403, 429} or "captcha" in response.text.lower()

    def process_response(self, request, response, spider):
        if self.is_retryable_response(response):
            return self.retry_with_new_proxy(request, spider) or response
        return response

    def process_exception(self, request, exception, spider):
        return self.retry_with_new_proxy(request, spider)

This is a minimal example, not a universal retry policy. It deliberately does not retry 407 responses because those usually require a credential, endpoint, or allowlist correction. Production middleware should add cooldowns, structured logs, and separate handling for network failures, authentication failures, and rejected pages. It can bind a proxy to a cookiejar for a workflow, route login traffic differently from public pages, and record the proxy identifier and retry reason for later review.

Enable this middleware in the project settings. Do not enable it alongside a proxy-pool middleware for the same requests unless you have defined and tested which component owns proxy assignment and retries.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CustomProxyMiddleware": 610,
}

Route requests by workflow, not only by domain

One spider can legitimately need more than one proxy behavior. The following policy is a useful starting point, subject to the site’s terms and the provider’s session rules.

For a sticky workflow, save the selected proxy in request metadata and use the same cookiejar ID for every request in that sequence. Release both when the sequence ends. This makes a session failure explainable: operators can see whether it was caused by credentials, the exit route, the application session, or the target response.

Configure proxy authentication correctly

Most 407 errors and early connection failures are endpoint or credential-format problems. Use the authentication model documented by the provider.

proxy = "http://user:password@ip:port"
request.meta["proxy"] = proxy

For providers that explicitly require Basic Auth in a header:

import base64

credentials = base64.b64encode(b"user:password").decode("ascii")
yield scrapy.Request(
    url,
    callback=self.parse,
    meta={"proxy": "http://ip:port"},
    headers={"Proxy-Authorization": f"Basic {credentials}"},
)

Check the scheme, port, special-character encoding, and whether the provider uses username/password or IP allowlisting. Do not send both URL credentials and a Proxy-Authorization header unless the provider supports that combination. Rola IP documents protocol support, proxy credentials, and whitelist access.

Before debugging a spider, make one request with the exact endpoint and authentication method. A failure at this stage is an integration problem, not a rotation problem. Also avoid writing full proxy URLs or credentials to logs; log a provider-defined identifier or a redacted host and port instead.

Make rotation stable

Use a new route for retryable failures

On a retryable failure, prefer a different exit. Keep retry ceilings low and record the reason: 403, 429, timeout, DNS or TLS error, challenge page, or unexpected redirect. Cool down an unhealthy proxy instead of immediately returning it to active rotation.

Do not layer multiple retry systems without deciding which one owns each event. For example, Scrapy’s RetryMiddleware, a rotating-proxies retry, and custom process_response logic can otherwise reschedule the same request several times. Define one owner for proxy-route retries and reserve the general retry mechanism for application-level transient errors.

Use sticky sessions for stateful workflows

Login flows, carts, cursor pagination, and multi-step forms normally need one proxy identity and one cookiejar until the workflow ends. Public listing pages can rotate more freely. Review the provider’s sessionid and sessiontime parameters before implementing sticky routing.

Size concurrency with the exit pool

Start at low concurrency, then adjust CONCURRENT_REQUESTS, DOWNLOAD_DELAY, DOWNLOAD_TIMEOUT, retry limits, and site-level limits from observed results. A planning estimate is:

minimum exits ≈ total requests per minute / requests per minute one IP can safely carry

The safe per-IP rate and buffer for retries must come from authorized testing, proxy quality, and the target site’s rules. They are not universal thresholds.

Verify the setup before scaling

1. Check that requests use the expected exit IP:

   scrapy shell "https://httpbin.org/ip"
   

   Repeat the request to distinguish a rotating gateway from a static endpoint. If exit country or rotation looks wrong, consult Rola IP’s IP attribution and switching guide.

2. Inspect logs to confirm the proxy identifier, retry count, failure class, and whether a retry selected a new route.

3. Run a small, authorized load test. Track 403 and 429 ratios, timeouts, average response time, per-proxy reuse, and session failures. Scale only after those metrics remain stable.

Minimum logging fields

Logs should make a failed request reconstructable without exposing credentials. Include a request ID, target host or route class, redacted proxy ID, cookiejar ID where relevant, retry count, response status, failure class, elapsed time, and the selected next action. This distinguishes a rejected page from an authentication error or a dead route.

Useful counters include live versus cooling-down proxies, retries by cause, unexpected-page ratio, and requests completed per route class. Alert on changes from the crawl’s own baseline. A fixed percentage threshold can be misleading when sites, geographies, and traffic patterns differ.

Troubleshooting

Connection timeout or refused connection

Temporarily cool down the proxy and check whether the route recovers. Avoid unlimited retries. Set alert thresholds from historical tests rather than fixed industry percentages.

meta["proxy"] is set, but traffic is direct

Check middleware order and search for later middleware that overwrites request.meta["proxy"]. This is common when retry, user-agent, and custom routing middleware all touch the request.

Repeated 403 or 429 responses

Changing IPs alone will not solve pacing, cookie, permissions, or browser-challenge problems. Review request frequency, session continuity, response validation, and the site’s allowed access methods. Use an available API instead of attempting to work around access controls.

It works locally but fails under load

Batch jobs increase exit reuse and replayed failures. Reproduce the production shape with a small controlled test before raising throughput.

Also compare production configuration with local configuration: environment variables, allowlisted source IP, endpoint region, middleware order, proxy pool size, and enabled retry middleware. A valid local test does not prove that those settings are identical in a deployment.

Conclusion

Choose a rotating gateway for low-maintenance deployment, a proxy pool for inventory-level control, or custom middleware for workflow-specific routing. Then verify exit selection, retry routing, and session behavior before increasing concurrency.

Rola IP can reduce gateway and exit-pool maintenance while offering geo and session options. Confirm current product parameters in the pricing and product page or test from the registration and login portal.

Compliance note: Use proxy rotation only for your own systems, authorized testing, or data access that follows site rules, robots rules, terms of service, and applicable law. Do not use it to circumvent access controls or account restrictions.