How to Scrape Google Search Results in 2026 (Python + Proxy Guide)

Why Google Scraping Got Harder in 2025

Understanding what changed matters because it tells you which approaches are wasting your time. In early 2025, Google shifted to requiring full JavaScript execution for standard search access, moving away from the pattern where a plain HTTP GET request returned a usable HTML response with embedded search results.

What this means practically: if you send a requests.get('https://www.google.com/search?q=test') today, you will receive an HTML response. But that response is a page shell with JavaScript includes, not a page with search results in it. The actual results load after JavaScript runs. Your scraper is getting a technically valid HTTP 200 response that contains no useful data. This is what makes it confusing, you do not get an error, you just get the wrong thing.

Google's detection system in 2026 evaluates four main signals: IP address reputation and origin, JavaScript execution capability (can the client actually run JS challenges?), TLS fingerprinting (does the TLS handshake look like a real browser or a Python library?), and behavioral signals (request timing, interaction patterns). Solving only some of these is not enough for reliable access. The thing that catches most scrapers is TLS fingerprinting: even with a real browser User-Agent header, the TLS handshake from a standard Python requests session is identifiably different from a Chromium-based browser.

The practical consequence: the three approaches below are structured around which detection signals each one addresses. googlesearch-python handles the basics well for low volume. Playwright handles the JS execution and TLS problem because it runs a real browser. Residential proxies handle the IP reputation problem for all three approaches.

Google's Search URL Parameters: What You Need to Know

Before writing any scraper code, understand the URL parameters. These control what results you get, how many, where they come from, and in what language. This is worth getting right before you scale.

The most important thing about geo-targeting: URL parameters alone are not sufficient. gl=de tells Google you want German results, but Google's system also weights results based on the IP address making the request. A request with gl=de coming from a US IP will produce different results than the same request from a German IP. For accurate country-specific SERP data, you need both the parameter and a proxy IP from that country. This is what actually means in practice for the target markets: US, UK, Germany, Indonesia, Korea, Japan, India, Netherlands, Hong Kong each need a matching proxy IP for accurate data.

from urllib.parse import urlencode

def build_google_url(query, country='us', lang='en', page=1):
    params = {
        'q': query,
        'gl': country,      # Country for results
        'hl': lang,         # Interface language
        'start': (page - 1) * 10,  # Pagination
        'ie': 'UTF-8',
        'oe': 'UTF-8',
    }
    return f'https://www.google.com/search?{urlencode(params)}'

# Examples for target markets
us_url   = build_google_url('proxy provider', country='us', lang='en')
de_url   = build_google_url('proxy anbieter', country='de', lang='de')
jp_url   = build_google_url('プロキシ',       country='jp', lang='ja')
kr_url   = build_google_url('프록시 서비스',  country='kr', lang='ko')
in_url   = build_google_url('proxy service', country='in', lang='en')

Approach 1: googlesearch-python Library (Small Projects)

For scripts that need Google results occasionally and do not need to run at high volume, the googlesearch-python library is the simplest option. It wraps the scraping logic and returns URLs as a Python generator. Install it with pip install googlesearch-python.

from googlesearch import search

# Basic usage - returns top 10 results for a query
for url in search("best residential proxies 2026", num_results=10):
    print(url)

# With a rotating proxy (recommended even at small scale)
proxy = 'http://username:[email protected]:31112'

for url in search(
    "web scraping tools",
    num_results=20,
    lang="en",
    proxy=proxy,
    ssl_verify=False  # Required when proxy handles SSL
):
    print(url)

# With language and region targeting
for url in search(
    "web scraping",
    num_results=10,
    lang="de",     # German results
    region="de",   # From Germany
    proxy=proxy,
    ssl_verify=False
):
    print(url)

The limitation is this: googlesearch-python returns only URLs, not titles or descriptions. If you need the full organic result data (rank position, title, meta description, domain), you need Approach 2 or 3. Use this library when you only need the URLs themselves, for example for a quick SEO backlink research script or a simple competitor URL discovery tool.

Approach 2: Playwright + Rotating Residential Proxies (Recommended)

This is the approach that actually holds up in 2026. Playwright launches a real Chromium browser, which handles JavaScript execution, TLS fingerprinting, and the various browser challenge checks that Google uses. Pair it with rotating residential proxies for IP-level protection and you have a scraper that addresses all four of Google's detection signals.

Install the dependencies:

pip install playwright beautifulsoup4
playwright install chromium

Complete Playwright SERP Scraper

import asyncio
import json
import time
from urllib.parse import urlencode
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup


# TorchProxies rotating residential credentials
PROXY_HOST = 'residential.torchproxies.com'
PROXY_PORT = '31112'
PROXY_USER = 'your_username'
PROXY_PASS = 'your_password'


def build_google_url(query, country='us', lang='en', page=1):
    params = {
        'q': query,
        'gl': country,
        'hl': lang,
        'start': (page - 1) * 10,
        'ie': 'UTF-8',
    }
    return f'https://www.google.com/search?{urlencode(params)}'


def parse_serp_html(html):
    """Parse Google SERP HTML with fallback selector strategies."""
    soup = BeautifulSoup(html, 'html.parser')
    results = []

    # Fallback selector strategies - Google changes HTML regularly
    selector_strategies = [
        {'container': '.tF2Cxc', 'title': 'h3.LC20lb', 'url': '.yuRUbf a', 'desc': '.VwiC3b'},
        {'container': '.g',       'title': 'h3',          'url': '.yuRUbf a', 'desc': '.VwiC3b'},
        {'container': '.MjjYud',  'title': 'h3',          'url': 'a',          'desc': "[data-sncf='1']"},
    ]

    for strategy in selector_strategies:
        containers = soup.select(strategy['container'])
        if len(containers) >= 3:  # Need at least 3 results to consider it valid
            for i, c in enumerate(containers, 1):
                title_el = c.select_one(strategy['title'])
                url_el   = c.select_one(strategy['url'])
                desc_el  = c.select_one(strategy['desc'])
                if title_el and url_el:
                    href = url_el.get('href', '')
                    if href.startswith('http'):  # Skip Google internal links
                        results.append({
                            'position': i,
                            'title': title_el.text.strip(),
                            'url': href,
                            'description': desc_el.text.strip() if desc_el else '',
                        })
            break  # Stop after first strategy that returns results

    return results


async def scrape_serp(query, country='us', lang='en', pages=1):
    all_results = []

    async with async_playwright() as pw:
        browser = await pw.chromium.launch(
            headless=True,
            args=[
                '--no-sandbox',
                '--disable-dev-shm-usage',
                '--disable-blink-features=AutomationControlled',
            ]
        )

        # One context per request - rotating proxy assigns fresh IP each time
        for page_num in range(1, pages + 1):
            context = await browser.new_context(
                proxy={
                    'server': f'http://{PROXY_HOST}:{PROXY_PORT}',
                    'username': PROXY_USER,
                    'password': PROXY_PASS,
                },
                # Match locale to target country
                locale=f'{lang}-{country.upper()}',
                timezone_id='America/New_York' if country == 'us' else 'Europe/Berlin',
            )

            # Remove webdriver flag
            await context.add_init_script(
                "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"
            )

            page = await context.new_page()
            url = build_google_url(query, country, lang, page_num)

            await page.goto(url, wait_until='networkidle', timeout=30000)
            await page.wait_for_timeout(2000)  # Let JS finish rendering

            html = await page.content()
            page_results = parse_serp_html(html)
            all_results.extend(page_results)

            await context.close()

            if page_num < pages:
                await asyncio.sleep(3)  # Delay between pages

        await browser.close()

    return all_results


# Run it
if __name__ == '__main__':
    results = asyncio.run(
        scrape_serp("residential proxy provider", country='us', lang='en', pages=2)
    )
    print(json.dumps(results, indent=2))
    print(f'\nScraped {len(results)} results')

A few things worth noting in this code. Creating a new browser context per request rather than reusing one context across all pages ensures that each request gets a fresh proxy IP assignment when using rotating proxies. The wait_for_timeout(2000) after page load gives JavaScript time to finish rendering the search results before we grab the HTML. Skipping this step is a common mistake and returns incomplete results.

The navigator.webdriver removal is important. Playwright sets this property to true by default, which is a reliable bot detection signal. Clearing it removes one detection vector. I have not personally tested every Google market with this specific configuration, so treat it as a starting point and adjust based on what you observe in production.

Geo-Targeting Google Results by Country in 2026

This is the section most guides skip. For rank tracking across multiple markets, the geo-targeting setup is the most operationally important part. Getting it wrong means your data is inaccurate regardless of how well your scraper handles detection.

TorchProxies Plan X provides 120M+ IPs across 195+ countries with city-level targeting available. For rank tracking across all ten of the above markets from a single proxy subscription, this is the practical solution. The alternative is managing ten separate proxy provider accounts for each country, which adds significant operational overhead.

Handling CAPTCHAs and Blocks

Google uses two main blocking mechanisms: soft blocks (CAPTCHA challenges that still return HTTP 200 but with a verification form) and hard blocks (HTTP 429 or 503 responses). Handling them correctly makes the difference between a scraper that recovers gracefully and one that silently returns garbage data for hours.

Detecting a CAPTCHA in Your Scraper

def is_captcha_page(html):
    """Check if Google returned a CAPTCHA page instead of results."""
    captcha_signals = [
        'Our systems have detected unusual traffic',
        'g-recaptcha',
        'recaptcha/api.js',
        'Sorry, we could not process your request',
    ]
    return any(signal in html for signal in captcha_signals)


def is_empty_serp(results):
    """Check if parsing returned no results (possible silent block)."""
    return len(results) == 0


# In your scraping loop:
html = await page.content()

if is_captcha_page(html):
    print("CAPTCHA detected. Rotating proxy and retrying after delay.")
    await context.close()
    await asyncio.sleep(10)
    # Retry with a new context (new proxy IP assigned automatically)
    continue

results = parse_serp_html(html)

if is_empty_serp(results):
    print("Empty results. Possible silent block or selector change.")
    # Log the HTML for inspection, do not silently continue

The empty results check is the more important of the two. CAPTCHAs are obvious. Silent blocks are invisible and dangerous for data quality. A scraper that returns zero results but continues running will produce a dataset full of missing entries that are hard to spot after the fact. Always validate that the parse returned a reasonable number of results before moving on.

Reducing CAPTCHA Frequency

• Use residential proxies, not datacenter proxies. Google's detection distinguishes between residential and datacenter IPs. Datacenter IPs trigger CAPTCHA significantly faster. The same query from a residential IP and a datacenter IP will hit CAPTCHA at completely different request rates.
• Add random delays between requests. Use random.uniform(2, 5) seconds between page requests rather than a fixed interval. Fixed intervals are more detectable as automated behavior.
• Rotate browser contexts between queries. A new browser context means a new browser fingerprint state, which resets session-level tracking. Creating a fresh context per query costs a few hundred milliseconds but significantly reduces fingerprint accumulation signals.
• Vary the User-Agent per context. Maintain a small pool of realistic Chrome user agent strings and randomly select one per browser context.

Fallback Selector Strategies for Google's Changing HTML

Google regularly changes its SERP HTML structure through A/B tests and platform updates. A scraper with a single hardcoded selector will break without warning. The fallback strategy in the code above is the right pattern, but it is worth explaining why each selector exists in the pool.

The validation check in the parser (if len(containers) >= 3) is deliberate. If a selector matches fewer than 3 containers, it is either the wrong selector or a non-results page. Falling through to the next strategy in the list handles Google's A/B test variants automatically without manual intervention.

When to Use a SERP API Instead (Anti-Commercial Honesty)

I want to be straightforward about this. There are situations where building and maintaining a DIY Google scraper is the wrong approach, and using a dedicated SERP API is the right one. This is worth getting right before you invest engineering time.

The honest calculation: a DIY scraper requires ongoing maintenance every time Google changes something, which happens frequently. I would not estimate less than a few hours per month in maintenance time for a production Google scraper in 2026. At low query volumes, that maintenance overhead is a significant cost relative to the scraping cost. At very high volumes, it is negligible. The break-even point depends entirely on your engineering cost and query volume.

TorchProxies is the proxy infrastructure layer regardless of which approach you choose. DIY scrapers need rotating residential proxies to function reliably against Google. SERP API providers run their own proxy infrastructure. If you build your own scraper, TorchProxies handles the IP rotation so you can focus on the parsing layer.

Common Mistakes When Scraping Google in 2026

The Bottom Line

Scraping Google search results in 2026 requires a browser-based approach. Simple HTTP requests do not work reliably anymore. Playwright with rotating residential proxies is the pattern that addresses all of Google's detection signals: JavaScript execution, TLS fingerprinting, IP reputation, and behavioral patterns. The URL parameter system gives you precise geo-targeting across every major market. Fallback selector strategies keep the scraper running through Google's regular HTML changes.

For small scripts, googlesearch-python with a proxy is enough. For production multi-market SERP tracking, the Playwright approach above is the right baseline. For very high volume commercial deployments, evaluate SERP APIs against the maintenance cost of running your own infrastructure before committing to a DIY build.

FAQs