.svg)
.svg)
.svg)
The Complete Guide to
Proxies for Web Scraping
2026
You want to scrape at scale. You keep getting blocked. The blog posts you have read all say "use residential proxies" but none tell you why your requests are failing or which type actually fits your use case. This guide fixes that. Every proxy type, the real reasons sites block you, rotation strategies that work, pricing decoded, and a clear decision framework. No fluff. No filler.
Why Proxies Are Non-Negotiable in 2026
In 2019, you could rotate a list of datacenter IPs and scrape most targets without problems. That era is gone.
The anti-bot industry has matured. Today's protection stacks, Cloudflare, Akamai, and PerimeterX, analyze signals you might not even know you are sending:
- Your TLS fingerprint (the cryptographic handshake your HTTP client makes)
- Your browser fingerprint (user agent, accepted encodings, header order)
- The IP reputation score and ASN (Autonomous System Number)
- Behavioral signals: request timing, mouse movement, scroll patterns
- Whether your IP has ever been seen on a residential network or only in a datacenter
This is why cheap datacenter proxies fail on any site using Cloudflare. They do not just check your IP. They check whether your IP has ever looked like a real person browsing from a real home.
The Four Proxy Types, Explained Honestly
Datacenter Proxies
IPs hosted in data centers, AWS, DigitalOcean, Hetzner. Fast, cheap, easy to scale. Also the most detectable type of proxy in existence.
Major sites like Amazon, LinkedIn, and most eCommerce platforms maintain blocklists of known datacenter ASNs. If your IP resolves to one of these blocks, the site blocks you before you send a single request.
Residential Proxies
Residential proxies route your traffic through real home IP addresses, actual people's internet connections sourced through opt-in networks. These IPs have real usage histories, belong to real ISPs, and look exactly like a person browsing from their couch.
Standard Residential: Real IPs, good trust scores, rotated on each request or on a timer. Excellent for most scraping tasks where you do not need to hold a session for more than a few minutes.
Premium Residential: Higher-quality IP pools, better success rates against harder targets, less abuse history, faster response times. The extra cost is justified when scraping high-protection sites at scale.
- For value, flexibility, and everyday scraping reliability. Choose Standard Residential
- When success rate, pool quality, and performance matter more than budget. Choose Premium Residential
ISP Proxies (Static Residential)
ISP proxies combine the trust score of a residential IP with the speed and stability of a datacenter connection. IP addresses are leased from real ISPs (AT&T, Windstream, RCN) but hosted on high-performance servers, not someone's home router.
The result: a static IP that never changes, looks 100% residential to anti-bot systems, runs at 10 Gbps, and can hold session state indefinitely.
X Residential: The Hybrid Approach
Torch's Plan X blends residential IPs for trust and anonymity with ISP IPs for speed and stability. Built for demanding, high-throughput scraping workflows where you cannot afford to choose between trustworthiness and performance.
Quick Decision Guide
| Proxy Type | Best For | Avoid When | Price Range |
|---|---|---|---|
| Datacenter | Public data, open APIs | Cloudflare, Amazon, social platforms | Lowest |
| Standard Residential | SEO, price monitoring, mid-tier targets | Long sessions needing persistent identity | Mid |
| ISP (Static Residential) | Long sessions, account management, high speed | When IP variety matters more than stability | Mid-Premium |
| Premium Residential / Plan X | High-protection targets, production scale | Budget is extremely tight | Premium |
| Mobile | Heaviest protection, last resort | Budget-conscious projects | Highest |
Why You Are Still Getting Blocked
1. You Are Not Rotating Headers
Websites do not just check your IP. They check everything. Your User-Agent, Accept-Encoding, Accept-Language, and the order these headers appear are all fingerprinting signals. If you rotate IPs but send identical headers every time, you are still identifiable.
- Rotate your User-Agent string with each request
- Match your headers to what a real browser sends for that User-Agent
- Header order matters for TLS fingerprinting
2. You Are Sending Requests Too Fast
Real humans do not request 200 pages per minute. Anti-bot systems track request velocity per IP. Even with rotation, hammering a target will trigger rate limits or soft blocks.
- Add randomized delays between requests (1.5s to 5s depending on target)
- Implement exponential backoff when you hit 429 or 503 responses
- Never retry failed requests immediately
3. Your Proxy Pool Has Abused IPs
This is the dirty secret the industry does not advertise. Large, cheap residential proxy pools often contain IPs that have been used and flagged across hundreds of scraping projects. When you get one of these IPs, you inherit their bad reputation.
4. Session Management Issues
If your target requires session continuity, cookies, login state, or basket state, rotating IPs mid-session triggers immediate detection. You need sticky sessions: same IP for the duration of a logical session, then rotation.
5. Cloudflare Bot Detection Has Upgraded
In 2026, Cloudflare's JS challenges require full browser execution. Turnstile CAPTCHAs now appear after just 3 to 5 requests from suspicious IPs. Raw residential proxies with Python requests will not cut it against Cloudflare.
IP Rotation Strategies That Work
IP rotation is not just "use a different IP each time." The right strategy depends on what you are scraping and how the target detects bots.
Per-Request Rotation
A new IP for every HTTP request. Most aggressive and most anonymous. Best for stateless scraping: product prices, search results, public listings where session continuity is not needed.
Sticky Sessions
The same IP held for a configurable window, typically 1, 10, or 30 minutes. Use this when your target requires session continuity: logging in, browsing a catalogue, completing a checkout flow. Most providers implement sticky sessions via a session ID parameter in the proxy URL.
Domain-Based Rotation
Assign different IPs or IP pools to different target domains. Prevents cross-domain contamination. If an IP gets flagged on Amazon, it is not immediately burned on eBay too.
Geo-Targeted Rotation
Route requests through IPs that match the geographic region of your target. If you are scraping UK pricing data, a US IP will get you US prices or get blocked entirely. City-level targeting is essential for localized data collection.
The Pricing Models Decoded
This is where most developers get burned. The $/GB pricing model seems straightforward until you start using it.
Bandwidth-Based Pricing ($/GB)
You pay for the data transferred through the proxy. The problem: bandwidth consumption is wildly variable depending on what you are scraping.
- A minimal JSON API response: ~2KB per request = 500,000 requests per GB
- A full HTML product page: ~50KB per request = 20,000 requests per GB
- A JS-heavy page via headless browser: ~500KB per request = 2,000 requests per GB
IP-Based Pricing ($/IP)
You pay for a fixed number of IPs, typically for a month. Bandwidth is unlimited or very high. This is the ISP proxy model and it is far more predictable for production workloads.
The Math That Changes Everything
| Provider Type | Listed Price | Success Rate | Effective Cost |
|---|---|---|---|
| Cheap provider | $5/GB | 60% | $8.33/GB equivalent |
| Premium provider | $10/GB | 98% | $10.20/GB equivalent |
The "cheap" option costs more per successful request. Every time.
How to Pick a Provider Without Getting Burned
Step 1: Know Your Target First
Low protection (public government sites, open APIs): Datacenter proxies are fine and significantly cheaper.
Medium protection (eCommerce, news sites with rate limiting): Standard or premium residential proxies.
High protection (Amazon, LinkedIn, social platforms, sneaker sites): Premium residential, ISP, or Plan X hybrid. Possibly add a headless browser layer.
Step 2: Test Before You Commit
Any reputable provider offers a trial. Use it. Test against your actual target, not a generic test URL. Measure success rate. Not speed. Not ping. Success rate.
Step 3: Ask About IP Pool Quality
- Are IPs ethically sourced through opt-in networks?
- How often is the pool cleaned of burned IPs?
- What is the ratio of unique IPs to bandwidth sold?
- Can you see the ASN distribution?
A good provider answers these questions directly. A bad one pivots to talking about pool size.
Step 4: Check Concurrent Connection Limits
Some providers silently throttle concurrent connections even on plans that claim to be unlimited. Test this during your trial by running your actual concurrency level.
Step 5: Evaluate Support Quality
At scale, something will go wrong. Test their response time and technical depth during your trial period, not after committing to a large plan.
Which Scraper Are You?
Most proxy problems fall into one of six patterns. Developers run into the same walls at the same stages. Find the profile that matches where you are right now and skip straight to what actually fixes it..
Technical Setup
The Proxy URL Format
http://username:[email protected]:port
http://username-session-abc123:[email protected]:port
socks5://username:[email protected]:port
http://gateway.provider.com:port
Python Integration
import requests
import time, random
proxies = {
'http': 'http://user:[email protected]:port',
'https': 'http://user:[email protected]:port',
}
def scrape_with_retry(url, max_retries=3):
for attempt in range(max_retries):
try:
# Randomize delays to mimic human behavior
time.sleep(random.uniform(1.5, 4.0))
response = requests.get(url, proxies=proxies, timeout=30)
if response.status_code == 200:
return response
except Exception as e:
wait = 2 ** attempt # Exponential backoff
time.sleep(wait)
return None
What to Monitor in Production
Legal and Ethical Considerations
Web scraping is legal in most jurisdictions when collecting publicly available data, content anyone can access without logging in. This was reinforced by the hiQ v. LinkedIn ruling in the US.
Where it gets complicated:
- Scraping content behind login walls or paywalls
- Violating a site's Terms of Service (civil, not criminal, in most cases)
- Collecting personal data in GDPR or CCPA jurisdictions
- Copyright-protected content: articles, images, product descriptions
Finding Your Setup with TorchProxies
| Your Situation | Torch Product | Key Advantage |
|---|---|---|
| First scraping project, general targets | Standard Residential | Value + free trial, no card needed |
| High-protection sites, eCommerce | Premium Residential | Cleaner pools, higher success rates |
| Long sessions, account management, sneaker bots | ISP Proxies | Static IPs, 10Gbps, unlimited bandwidth |
| Production-scale, demanding pipelines | Plan X Hybrid | Residential trust + ISP speed |
All Torch plans include:
- City-level geo-targeting across 190+ countries
- Rotating and sticky session modes
- API access and dashboard management
- Free trial, no credit card required
- Discord and email support