Web Scraping Best Practices in 2026

Why Scrapers Get Blocked in 2026

Before getting into practices, it helps to understand what you are actually up against. Detection does not happen after you have downloaded a page. It happens in the first few milliseconds of the connection, before any HTML is served. Modern anti-bot systems run multiple detection layers simultaneously, and bypassing one while failing another still gets you blocked.

Proxy Tier Selection: Matching IP Type to Target

This is the decision with the highest leverage. Choosing the wrong proxy type does not just reduce your success rate. On well-protected targets, the wrong proxy type produces a zero percent success rate regardless of everything else you do. The decision is not which proxy is best in general. It is which proxy type the target's anti-bot system will accept.

What this actually means in practice: if you are targeting Amazon, Shopify storefronts, or major footwear sites, datacenter proxies will not get you past the first layer of protection. A reviewer on Trustpilot confirmed this from experience with Plan X: "IPs test clean on whoer.to. Suitable for almost every use case." The clean fraud score is what matters on protected targets, not the proxy type label.

The Case for Target-Specific Pools

Most proxy guides stop at proxy type selection. What they skip over is that the same residential IP pool can have wildly different success rates depending on whether those IPs have been recently burned on the specific target you are hitting. This is where target-specific pools become relevant.

TorchProxies offers pre-configured pools for specific high-protection retail targets: Footsites (US, CA, EU, AU, SG, MY), Nike (US, EU, MY), Supreme (US, EU, JP), Yeezy Supply (US), Popmart (US, CA, EU, AU, SG, MY, JP), and Pokemon Center (US, CA, EU, AU, SG, MY, JP). These pools contain IPs that have been validated against those specific targets. Most providers make you figure out the right configuration yourself, which on protected retail targets usually means burning through a lot of trial and error before finding IPs with acceptable success rates.

Session Management: Sticky vs. Rotating

This one catches teams constantly. The mistake is picking either sticky or rotating and applying it to everything. The right answer is different for different task types, and using the wrong one will break your scraper in ways that are genuinely hard to debug.

Rotating Proxies: When and How

Use rotating proxies when each page you are collecting is independent of every other page. Product listings, public pricing pages, news articles, search results. No cookies need to persist between requests. New IP per request means no single address accumulates enough request history to trigger rate limiting or reputation scoring.

Where I see this fail: teams using rotating proxies on flows that actually require session continuity. If you hit page 1 of a paginated result from one IP and then hit page 2 from a different IP, many sites will detect this as session anomalous behavior. You will get served a captcha or a redirect to the homepage, not a useful 429 response. The scraper logs look fine. The data is empty.

Sticky Sessions: When and How

Use sticky sessions for any task that requires maintaining state across requests: login flows, authenticated pages, checkout sequences, anything where the site is tracking your session via cookie. The IP and the cookie jar must stay consistent for the duration of the session. If you rotate the IP while keeping the same cookies, the site sees a session originating from two different network locations, which is a strong bot signal.

TorchProxies supports sticky sessions on all plans, with session duration configurable through the dashboard. For ISP static proxies specifically, the IP remains fixed indefinitely, which makes them ideal for account management tasks where you need the same identity over days or weeks rather than just a single session.

Rate Limiting and Request Timing

Sending 100 requests per second from a residential IP is technically possible. It is also exactly the kind of behavior that gets that IP flagged within minutes. Rate limiting is not about being slow. It is about being indistinguishable from a human.

Human Timing Patterns

Real users do not browse pages every 2.0 seconds. They read, scroll, click around, pause, and navigate non-linearly. Fixed-interval requests, even generous ones, are detectable by timing analysis. Use randomized delays drawn from a realistic distribution rather than uniform sleeps.

A conservative starting point: 1 to 5 requests per minute per IP on smaller sites. Well-resourced platforms can tolerate 20 to 30 per minute before flagging, according to ScrapeHero's analysis. The key is that these numbers need to be varied, not constant. A scraper that makes exactly 12 requests per minute, every minute, is as detectable as one that makes 1,200.

Exponential Backoff with Jitter

Standard exponential backoff is a well-known pattern, and sophisticated bot detection systems have started recognizing it as a detection signal in itself. The deterministic nature of pure exponential backoff, always doubling, is its own fingerprint. Add random jitter to break the pattern:

import time, random

def backoff_with_jitter(attempt, base=2, cap=60):
    # Exponential growth with a random offset
    wait = min(cap, (base ** attempt)) + random.uniform(0, 1)
    time.sleep(wait)

def fetch_with_backoff(url, session, proxies, max_retries=5):
    for attempt in range(max_retries):
        response = session.get(url, proxies=proxies, timeout=30)
        if response.status_code == 200:
            return response
        elif response.status_code == 429:
            retry_after = int(response.headers.get('Retry-After', 2 ** attempt))
            time.sleep(retry_after)
        elif response.status_code in (403, 503):
            # Rotate IP and back off before retry
            backoff_with_jitter(attempt)
    return None

When You Do Not Need Proxies

I will not go deep on proxy-free scraping targets here because it deserves its own guide. But the short version: public government databases, academic data repositories, official open APIs, and any site without IP-based rate limiting or bot detection do not need proxies. Adding proxy infrastructure to everything is a cost you are paying unnecessarily. Always test without proxies first on low-risk targets. You only need proxies when the target actively requires them.

Request Headers and TLS Fingerprinting

Getting the IP right and the timing right still leaves two major detection vectors: request headers and TLS fingerprinting. These are the ones most guides skip over, and in 2026 they are increasingly the primary reason properly proxied scrapers still get blocked.

Headers: Complete or Caught

A real browser sends a full set of HTTP headers on every request: User-Agent, Accept, Accept-Language, Accept-Encoding, Connection, and often Referer. Python's requests library sends minimal headers by default. Any site examining incoming headers will immediately distinguish a Python scraper from a browser.

Set a complete, browser-consistent header set. The User-Agent string is the most commonly discussed, but the full header consistency matters more than any single header. Use a current browser version. A User-Agent referencing Chrome 2021 is a statistical anomaly that real traffic does not produce in 2026.

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1"
}

TLS Fingerprinting: The 2026 Problem

This is the part most guides do not cover, and it is increasingly the primary detection mechanism on heavily protected targets in 2026. During the HTTPS handshake, your scraper sends a TLS ClientHello message that contains a specific combination of cipher suites and extensions. Python's underlying OpenSSL stack produces a fingerprint that differs from Chrome, Firefox, or Safari. Detection systems identify this before any of your headers are read.

The practical fix: use curl_cffi, which impersonates the TLS handshake of real browsers. It is a drop-in replacement for requests on most scraping tasks.

from curl_cffi import requests as cffi_requests

response = cffi_requests.get(
    "https://target.com/product-page",
    impersonate="chrome124",
    proxies={
        "http": "http://user:[email protected]:31112",
        "https": "http://user:[email protected]:31111"
    }
)

What this actually means in practice: on Cloudflare-protected targets, switching from requests to curl_cffi with a residential IP often converts a 100% block rate to a working scraper, with no other changes. The TLS fingerprint is the thing that was failing, not the IP quality.

Geolocation Matching: Why It Matters for Your Target Markets

Your proxy IP's geolocation needs to match what the target site expects to see. A US e-commerce site receiving traffic from a German IP on a US-only promotion is an immediate anomaly signal. This is not just about bypassing geo-restrictions on content. It is about making your traffic statistically plausible.

TorchProxies supports 195 countries with city-level targeting on supported plans, and offers state-level targeting for US IPs on select pools. The EU sub-regional pools (EU1 through EU5, EU SC Nordic, EU W Benelux and France, EU S Southern, EU C Central, EU E Eastern) give more precise regional control than a generic "Europe" pool option.

Legal Compliance in 2026: What You Actually Need to Know

I am not a lawyer. None of what follows is legal advice. But the legal landscape has shifted enough in 2026 that ignoring it is a meaningful operational risk, not just an ethical concern.

The Current Legal Framework

The hiQ v. LinkedIn Ninth Circuit ruling established that accessing publicly available data does not constitute unauthorized access under the US Computer Fraud and Abuse Act (CFAA). This is the strongest legal precedent for scraping public web data. Publicly available means visible without authentication. The moment you bypass a login, paywall, or any access control mechanism, the CFAA risk increases significantly.

The GDPR applies to any personally identifiable information collected from EU users, regardless of where your scraper runs. Names, email addresses, behavioral data tied to identifiable individuals all require a lawful basis for collection. In practice, this means most production scraping operations should be designed to collect factual data (prices, product specs, business listings) rather than personal data.

A proposed US law, the AI Accountability for Publishers Act introduced in February 2026, would make robots.txt legally enforceable for AI training data collection. Even before it potentially passes, treating robots.txt as a compliance requirement rather than an advisory is the safer posture.

One Compliance Note Most Guides Skip

Even when scraping is legal, how you do it matters for legal risk. Aggressive scraping that causes measurable server degradation can be characterized as a denial-of-service attack under the CFAA and equivalent laws in other jurisdictions, regardless of whether the underlying data collection is lawful. Rate limiting is not just a detection avoidance technique. It is part of compliant scraping behavior.

The Most Common Mistakes at Scale

What I see fail repeatedly, after watching teams build and debug scraping pipelines over the last few years.

Which TorchProxies Plan for Which Scraping Use Case

Picking a plan is simpler than most articles make it. The question is not which plan is most powerful in general. It is which IP quality your specific target needs to return real content.

All plans are pay-as-you-go with no long-term contracts and no rate limits. The 10 Gbps network with automatic retries means your throughput is not capped at the infrastructure level. A free trial is available on all products with no credit card required, which lets you test your specific target before committing to a plan.

Final Verdict

The fundamentals of web scraping have not changed: send requests, parse responses, store data. What has changed is the sophistication of the detection systems sitting between your scraper and the HTML it wants. In 2026, getting blocked is rarely a code problem. It is an infrastructure problem, a session management problem, or a timing problem.

The practices that actually move the needle: pick the right proxy tier for the target's protection level, match session type to task type, randomize request timing, set complete browser-like headers, use TLS fingerprinting countermeasures on protected targets, and validate data content rather than just structural success.

On the question of whether you need proxies at all: only when the target requires them. Proxy infrastructure adds cost and complexity. Add it precisely where it solves a concrete problem, not as a default for every project.

Frequently Asked Questions