.svg)
.svg)
.svg)
Scraping Amazon
with Python in 2026:
Complete Guide
- Datacenter IPs are blocked at the network layer before a single header is read. Residential proxies with consumer ISP ASNs are the baseline requirement, not an upgrade.
- Chrome sends 12 headers in a specific order. Python sends 2. That gap triggers Layer 3 detection within 20-50 requests. Adding a User-Agent alone does not fix it.
- AWS WAF added JA4 TLS fingerprinting in March 2025. Python's
requestslibrary is now identifiable at the handshake layer before any header is read. - Most Amazon product pages do not need Playwright. Title, price, and availability are in the initial HTML. Use async httpx for scale.
- Single selectors break silently. Amazon A/B tests its UI constantly. Fallback chains are required, not optional.
- The 503 is an IP problem first, a code problem second. Fix the proxy layer before debugging your parser.
You have probably read a guide that showed you three lines of code, maybe a static User-Agent header, and claimed that was enough to scrape Amazon. And then you ran it, and it worked for exactly long enough to feel confident before it stopped working entirely.
Here's the thing: scraping Amazon with Python is solvable. But it requires understanding what is actually happening when your requests get blocked. The 503 is not random. Amazon's infrastructure is built on AWS, and the same AWS WAF Bot Control system they sell to enterprises to protect their own sites is what protects Amazon's product pages. Knowing that changes how you approach every decision in your scraper.
This guide covers the full picture: why the blocks happen, what Python tools actually work and when, complete production-ready code for product pages, search results, and reviews, and how to keep it running at scale.
- 1Whether scraping Amazon is legal and where the line is
- 2Why Amazon keeps blocking your scraper: the 4-layer detection model
- 3Which Python tool to use for each page type
- 4The exact header stack Chrome sends, copy-paste ready
- 5How to choose the right proxy type with verified success rate data
- 6Production-ready code: product pages, search results, async scaling, Playwright
- 7Every failure mode and its fix
Is Scraping Amazon Legal? Answer This Before Running Code
Scraping publicly visible Amazon data is legal in most jurisdictions. Amazon's ToS prohibits it as a contractual matter, not a criminal one. The 9th Circuit's 2022 hiQ v. LinkedIn ruling confirmed that accessing publicly available data does not violate the CFAA. Never scrape login-protected pages, personal buyer data, or internal APIs.
Terms of Service Violation vs. Breaking the Law
Amazon's Conditions of Use prohibit "crawling, scraping, data-mining" the site. Violating a Terms of Service is a contractual breach, not a crime. The consequence is that Amazon can terminate your access, block your IPs, or potentially pursue a civil claim. It is not the Computer Fraud and Abuse Act violation that aggressive cease-and-desist letters sometimes imply.
The US 9th Circuit Court of Appeals ruled in 2022 in the hiQ Labs v. LinkedIn case that scraping publicly available data does not constitute unauthorized access under the CFAA. The court's reasoning was that data publicly visible to any unauthenticated visitor cannot be "protected" in the sense the statute requires. This ruling is the clearest case law on public web scraping as of 2026 and is widely cited by legal practitioners in this space.
What You Should Never Scrape
The legal picture changes entirely the moment you cross into protected territory. These are hard stops regardless of jurisdiction:
- Any page requiring Amazon account login (order history, saved lists, seller dashboards)
- Personal buyer or seller data that is not publicly displayed
- Internal Amazon APIs, including those used by the mobile app that are not publicly documented
- Data that would enable re-identification of individual consumers
What About Amazon's Product Advertising API?
Amazon offers the Product Advertising API 5.0 (PAAPI), which provides structured product data for approved affiliates. If you are building a comparison tool or affiliate product page and already participate in Amazon Associates, PAAPI is worth evaluating before you write a scraper. The API has rate limits and requires affiliate approval, but it is the path of least resistance for structured product data at low-to-medium volume. Scraping becomes the right answer when you need data the API does not expose, need it at volumes or frequencies the API does not support, or are not an Amazon affiliate.
Why Amazon Keeps Blocking Your Scraping Requests
Most guides stop at "Amazon detects bots." That is accurate and useless. Amazon uses AWS WAF Bot Control which runs four detection layers in sequence: IP/ASN reputation, TLS fingerprinting (JA3/JA4), HTTP request fingerprinting, and behavioral ML scoring. Adding a User-Agent only patches part of Layer 3 and does nothing about the other three. Here is what is actually happening at the network layer, in the order Amazon evaluates it.
requests has a known JA3 signature. AWS WAF added JA4 fingerprinting in March 2025, making Python's HTTP clients even more identifiable than they were before.requests sends 4 by default. Beyond header presence, the order headers appear in the request is a detection signal. Adding a User-Agent addresses one header but leaves the order mismatch and the missing Sec-Fetch-* headers intact. This is why "just add a User-Agent" works briefly then fails.Why Running Your Scraper on AWS or GCP Makes It Worse
If your Python scraper runs on an EC2 instance, a Google Cloud VM, or any major cloud compute platform, you are scraping Amazon from IPs that belong to Amazon's own ASN neighbors. Amazon's WAF has explicit rules for these address ranges. You are not just blocked at Layer 1 - you are pre-burned. Always run scrapers behind residential proxies, never from cloud servers directly.
Which Python Tool Should You Actually Use?
Amazon has different page types with different rendering behaviors. Matching your tool to the page type saves hours of debugging: requests + BeautifulSoup for static product data, httpx + asyncio for bulk scale, Playwright for lazy-loaded reviews or scroll-based search results only.
| Use Case | Best Tool | Proxy Needed? | JS Rendering? |
|---|---|---|---|
| Product detail page (title, price, rating) | requests + BeautifulSoup | Yes (residential) | No - data is in initial HTML |
| Bulk ASIN list, 100+ products | httpx async | Yes (rotating residential) | No |
| Product reviews (lazy-loaded) | Playwright | Yes (residential) | Yes |
| Search results with infinite scroll | Playwright | Yes (residential) | Sometimes |
| Price monitoring pipeline at scale | httpx async + residential proxies | Yes (premium rotating) | No |
The Complete Header Stack That Passes Amazon's Layer 3 Check
The single most actionable fix for most developers hitting the 20-request wall. You need 12 headers in Chrome-matching order. Header order matters as much as header content. Always use requests.Session() to preserve cookies across requests.
# This header stack mirrors Chrome 122 on Windows 10.
# Keep the key order exactly as written.
# Python 3.7+ dicts preserve insertion order.
import requests
import time, random
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/122.0.0.0 Safari/537.36"
),
"Accept": (
"text/html,application/xhtml+xml,application/xml;"
"q=0.9,image/avif,image/webp,image/apng,*/*;"
"q=0.8,application/signed-exchange;v=b3;q=0.7"
),
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Cache-Control": "max-age=0",
}
# Always use a Session. Amazon sets cookies on the first request.
# Dropping those cookies resets your session trust score.
session = requests.Session()
session.headers.update(HEADERS)
Why Sec-Fetch Headers Matter Specifically
The Sec-Fetch-* headers were introduced in Chrome 76 and are sent by every Chromium-based browser on every navigation. They tell the server the context of the request: is this a top-level navigation? Is it from the same site? Was it user-initiated? Amazon's WAF specifically checks for Sec-Fetch-Dest: document and Sec-Fetch-Site: none on direct URL navigations. Requests missing these headers are trivially identifiable as non-browser traffic.
The Proxy Layer: Why Datacenter Fails on Amazon and What to Use
Once you understand the four detection layers, the proxy choice is straightforward. You need an IP whose ASN passes Layer 1, which means a consumer ISP ASN, not a datacenter ASN. Datacenter proxies hit 20-40% success rates on Amazon product pages because their ASN is flagged before a single header is processed. Residential proxies reach 85-95% with correct headers.
Success Rate Reality by Proxy Type on Amazon
| Proxy Type | Amazon Product Pages | Search Results | Review Pages | Session / Login |
|---|---|---|---|---|
| Datacenter | 20-40% | 15-35% | 10-25% | Near 0% |
| Residential (rotating) | 85-95% | 80-92% | 78-90% | Moderate |
| ISP (static residential) | 88-95% | 85-92% | 83-91% | High (stable IP) |
Success rate ranges sourced from independent proxy benchmarks: Proxyway Proxy Market Research 2024 and Bright Data proxy comparison (2025). Rates reflect correct header configuration; results vary by IP pool quality and request rate.
Rotating vs. Sticky Sessions for Amazon
Rotating residential proxies assign a new IP on each request or on a configurable timer. This is correct for stateless scraping where you do not need to maintain session context between requests. But here's the thing: several Amazon page types do require session continuity.
Review pages, "Customers also bought" sections, and any page that requires cookie state from a prior navigation need a consistent IP for the duration of that logical session. Rotating mid-session resets your trust score and triggers re-evaluation at Layer 4. The rule is: rotate between distinct scraping tasks, not within a session.
requests.Session() with residential proxy credentials, exponential backoff on 503, and a human-like delay between requests. This is the production pattern, not the tutorial pattern.
import requests, time, random
from typing import Optional
class AmazonScraper:
def __init__(self, proxy_user: str, proxy_pass: str):
self.session = requests.Session()
self.session.headers.update(HEADERS) # HEADERS from Section 4
self.proxies = {
"http": f"http://{proxy_user}:{proxy_pass}@residential.torchproxies.com:8080",
"https": f"http://{proxy_user}:{proxy_pass}@residential.torchproxies.com:8080",
}
self.session.proxies.update(self.proxies)
def get(self, url: str, retries: int = 3) -> Optional[requests.Response]:
for attempt in range(retries):
# Human-like delay: 2-5 seconds between requests
time.sleep(random.uniform(2.0, 5.0))
try:
resp = self.session.get(url, timeout=15)
if resp.status_code == 200:
return resp
if resp.status_code == 503:
# Exponential backoff on 503
time.sleep((2 ** attempt) + random.uniform(0, 1))
continue
except requests.RequestException as e:
if attempt == retries - 1:
raise
return None
Scraping Amazon Product Pages: Full Working Code
Setup
pip install requests beautifulsoup4 lxml httpx
Parsing a Product Detail Page
The code below extracts title, price, rating, review count, and availability from any Amazon product page URL. Every field uses a fallback chain so that when Amazon changes its HTML in an A/B test, you get None instead of a crash.
from bs4 import BeautifulSoup
def extract_product(html: str) -> dict:
soup = BeautifulSoup(html, "lxml")
# Title: ID selector is stable. Class-based titles change frequently.
title_el = (
soup.select_one("#productTitle")
or soup.select_one("#title span")
or soup.select_one("h1.a-size-large span")
)
title = title_el.get_text(strip=True) if title_el else None
# Price: Amazon shows price in multiple elements depending on context
price_el = (
soup.select_one(".a-price .a-offscreen")
or soup.select_one("#priceblock_ourprice")
or soup.select_one("#priceblock_dealprice")
or soup.select_one('[data-a-color="price"] .a-offscreen')
)
price = price_el.get_text(strip=True) if price_el else None
# Rating
rating_el = (
soup.select_one('span[data-hook="rating-out-of-text"]')
or soup.select_one("i.a-icon-star span.a-icon-alt")
)
rating = rating_el.get_text(strip=True) if rating_el else None
# Review count
reviews_el = (
soup.select_one("#acrCustomerReviewText")
or soup.select_one('span[data-hook="total-review-count"]')
)
reviews = reviews_el.get_text(strip=True) if reviews_el else None
# Availability
avail_el = soup.select_one("#availability span")
availability = avail_el.get_text(strip=True) if avail_el else None
return {
"title": title,
"price": price,
"rating": rating,
"review_count": reviews,
"availability": availability,
}
# Use the AmazonScraper from Section 5 to get the HTML
def scrape_asin(scraper, asin: str) -> dict:
url = f"https://www.amazon.com/dp/{asin}"
resp = scraper.get(url)
if resp is None:
return {"asin": asin, "error": "request_failed"}
data = extract_product(resp.text)
data["asin"] = asin
return data
a-color-base.a-text-bold change on every deploy cycle. Never write a scraper with a single class-based selector and no fallback. ID selectors and data-hook attribute selectors are Amazon's most stable targeting points.
Exporting to CSV and JSON
import json, csv
def save_json(data: list[dict], path: str = "amazon_products.json"):
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=2)
def save_csv(data: list[dict], path: str = "amazon_products.csv"):
if not data: return
with open(path, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=data[0].keys())
writer.writeheader()
writer.writerows(data)
Scaling Up: Async Scraping with httpx
Running 500 product pages synchronously with a 3-second delay between each takes over 25 minutes. The same list with 5 concurrent async requests takes around 5 minutes. The trick is keeping concurrency within limits that do not trigger behavioral scoring.
import asyncio, httpx, random
from typing import List
# Reuse HEADERS from Section 4
async def scrape_asin_async(
client: httpx.AsyncClient,
asin: str,
sem: asyncio.Semaphore
) -> dict:
url = f"https://www.amazon.com/dp/{asin}"
async with sem:
# Randomised delay inside the semaphore slot
await asyncio.sleep(random.uniform(2.0, 5.0))
try:
resp = await client.get(url, timeout=15)
if resp.status_code == 200:
data = extract_product(resp.text) # from Section 6
data["asin"] = asin
return data
return {"asin": asin, "error": f"http_{resp.status_code}"}
except httpx.RequestError:
return {"asin": asin, "error": "request_error"}
async def scrape_batch(asins: List[str], proxy_url: str) -> List[dict]:
# Max 5 concurrent requests: safe for rotating residential pools
sem = asyncio.Semaphore(5)
async with httpx.AsyncClient(
headers=HEADERS,
proxies={"all://": proxy_url},
follow_redirects=True,
) as client:
tasks = [scrape_asin_async(client, asin, sem) for asin in asins]
return await asyncio.gather(*tasks)
# Entry point
if __name__ == "__main__":
PROXY = "http://user:[email protected]:8080"
asins = ["B098FKXT8L", "B07ZPKN6YR", "B09G9HD6PD"]
results = asyncio.run(scrape_batch(asins, PROXY))
save_json(results)
Scraping Amazon Search Results and Building a Product Pipeline
Most scraping guides only show product detail pages. A real workflow starts with a search query and works down to individual products. Use the [data-asin] attribute selector on search result pages. It is Amazon's most stable targeting point and survives HTML restructuring.
Amazon Search URL Parameters
# Base URL for Amazon search results
https://www.amazon.com/s?k=SEARCH+TERM&page=2
# Key parameters:
k # Search query (URL-encoded)
page # Page number (typically 1-20, about 400 total results max)
rh # Refinement: category node, price range, star rating
s # Sort: relevanceblender, price-asc-rank, review-rank
# Build search URLs in Python
import urllib.parse
def build_search_url(query: str, page: int = 1) -> str:
params = urllib.parse.urlencode({"k": query, "page": page})
return f"https://www.amazon.com/s?{params}"
Extracting ASINs and Product Data from Search Pages
from bs4 import BeautifulSoup
def parse_search_results(html: str) -> list[dict]:
soup = BeautifulSoup(html, "lxml")
products = []
# data-asin is the most stable selector on search pages.
# It is present on every product card and survives redesigns.
items = soup.select("[data-asin]:not([data-asin=''])")
for item in items:
asin = item.get("data-asin")
title_el = item.select_one("h2 a span") or item.select_one("h2 span.a-text-normal")
price_el = item.select_one("span.a-price .a-offscreen")
rating_el = item.select_one("span.a-icon-alt")
count_el = item.select_one("span.a-size-base.s-underline-text")
products.append({
"asin": asin,
"title": title_el.get_text(strip=True) if title_el else None,
"price": price_el.get_text(strip=True) if price_el else None,
"rating": rating_el.get_text(strip=True) if rating_el else None,
"review_count": count_el.get_text(strip=True) if count_el else None,
})
# Remove cards without a title (ads, banners, empty slots)
return [p for p in products if p["title"]]
Once you have a list of ASINs from the search results, feed them directly into the scrape_batch() function from Section 7 to pull full product detail data. This two-stage approach separates the rate strategies: you can hit search pages less frequently (they are heavier pages and more closely monitored) while running product detail pages at your full async concurrency.
When You Actually Need Playwright
Use Playwright for review pages with lazy loading, search results that require scrolling to trigger JavaScript rendering, or any page where the content you need is not in the initial HTML. Do not use it for standard product detail pages. Adding 150-300MB RAM and 1-2 seconds per launch for data that is already in the static HTML is waste. Note: even Playwright is detectable without additional configuration. The setup below addresses the most common detection signals.
from playwright.sync_api import sync_playwright
import time, random
def scrape_reviews_playwright(asin: str, proxy: dict) -> list[dict]:
with sync_playwright() as p:
browser = p.chromium.launch(
headless=True,
proxy=proxy,
args=["--disable-blink-features=AutomationControlled"],
)
ctx = browser.new_context(
user_agent=(
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/122.0.0.0 Safari/537.36"
),
viewport={"width": 1920, "height": 1080},
locale="en-US",
)
page = ctx.new_page()
# Block images/fonts to reduce bandwidth and fingerprint noise
page.route("**/*.{png,jpg,jpeg,gif,svg,woff,woff2}", lambda r: r.abort())
url = f"https://www.amazon.com/product-reviews/{asin}"
page.goto(url, wait_until="domcontentloaded", timeout=30000)
time.sleep(random.uniform(1.5, 3.0))
reviews = []
cards = page.locator('[data-hook="review"]').all()
for card in cards:
body_el = card.locator('[data-hook="review-body"]')
title_el = card.locator('[data-hook="review-title"]')
rating_el = card.locator('[data-hook="review-star-rating"]')
reviews.append({
"title": title_el.text_content() if title_el.count() > 0 else None,
"rating": rating_el.text_content() if rating_el.count() > 0 else None,
"body": body_el.text_content() if body_el.count() > 0 else None,
})
browser.close()
return reviews
# Proxy config for Playwright
proxy_config = {
"server": "http://residential.torchproxies.com:8080",
"username": "YOUR_USERNAME",
"password": "YOUR_PASSWORD",
}
pip install playwright, run playwright install chromium to download the browser binary. Headless mode works on servers but requires the system dependencies listed in Playwright's documentation for Linux environments. The --disable-blink-features=AutomationControlled flag removes the navigator.webdriver property that headless browsers expose by default.
Troubleshooting: Every Failure Mode and Its Fix
Check the response status code and body content first. A 503 with no body means Layer 1 or 2. A 200 with a "Robot Check" page means Layer 3. A 200 with prices missing means JavaScript rendering. Match your symptom below and skip straight to the fix.
data-a-color="price" as the selector fallback.
Scraping Amazon with Python: The Approach That Actually Holds
Scraping Amazon with Python is not a solved problem you can copy from a three-year-old tutorial. Amazon's detection runs four independent layers, and a fix that patches one layer while leaving the others untouched only buys you another 20 requests before the pattern accumulates again.
The approach that holds at scale is not about clever code. It is about matching the right tool to the right layer: residential proxies at the IP layer, the complete header stack at the HTTP layer, human-like timing at the behavioral layer, and selector fallback chains at the parsing layer. Get those four things right and most of the frustration goes away.
For production price monitoring or market intelligence workflows that need reliable throughput, TorchProxies Premium Residential handles the proxy layer so your engineering time goes into the data pipeline, not connection management. For login-based or session-persistent workflows, ISP Proxies give you the stable static IP identity that rotating pools cannot provide.
For a deeper look at the fingerprint detection layer (Layer 2) and how TLS impersonation fits into a full anti-detection stack, the companion guide on how to avoid fingerprint detection covers the JA3/JA4 mechanism and the tool landscape in detail.