Proxy Setup for LangChain, CrewAI and Claude Agents: A Developer's Guide (2026)

The Two Proxy Problems You're Actually Dealing With

Before touching any configuration, understand what you're actually routing. AI agent pipelines make two distinct types of outbound requests, and they need proxy configuration in different places.

Most developers hitting proxy issues are dealing with the tool request problem: their agent's WebBaseLoader can't load pages because the datacenter IP got blocked, or DuckDuckGoSearchRun starts returning empty results after 50 queries because it's rate-limiting the IP. That's a residential rotating proxy problem. Nothing to do with how you're calling Claude or GPT-4.

Corporate network restrictions are the LLM API call problem. You're behind a firewall that blocks direct connections to api.anthropic.com. That's a different configuration entirely.

Know which problem you have before picking a solution.

LangChain Proxy Setup

LangChain's tools and document loaders use Python's requests library for most HTTP calls. That means they respect HTTP_PROXY and HTTPS_PROXY environment variables natively. The catch: they read these variables at import time in some components, not at request time. Set them before your imports.

Method 1: Environment Variables (Global)

import os

# Set proxy BEFORE importing any LangChain components
os.environ["HTTP_PROXY"]  = "http://username:password:[email protected]:6011"
os.environ["HTTPS_PROXY"] = "http://username:password:[email protected]:6011"

# Now import LangChain
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_community.document_loaders import WebBaseLoader
from langchain_openai import ChatOpenAI

This covers DuckDuckGoSearchRun, BingSearchAPIWrapper, GoogleSearchAPIWrapper, and most other tools that use requests under the hood. It's the lowest-friction approach for development.

Method 2: WebBaseLoader with requests_kwargs

This is the cleaner approach for production. Instead of polluting env vars globally, pass proxy configuration directly to the loader. It also means different loaders can use different proxies, which matters when you're hitting geo-locked content and need region-specific IPs.

from langchain_community.document_loaders import WebBaseLoader

proxy_config = {
    "http":  "http://username:password:[email protected]:6011",
    "https": "http://username:password:[email protected]:6011",
}

loader = WebBaseLoader(
    web_paths=["https://example.com/page"],
    requests_kwargs={
        "proxies": proxy_config,
        "timeout": 30,
        "headers": {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
        }
    }
)

docs = loader.load()
print(docs[0].page_content[:500])

SOCKS5 Proxy in LangChain

TorchProxies supports SOCKS5 via the socksgeox.torchproxies.com host on port 6014. SOCKS5 routes all TCP traffic, not just HTTP, which is useful for tools that make non-HTTP requests. The setup is almost identical, but you need the requests[socks] package.

pip install requests[socks]

import os

os.environ["HTTP_PROXY"]  = "socks5://username:password:[email protected]:6014"
os.environ["HTTPS_PROXY"] = "socks5://username:password:[email protected]:6014"

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://example.com")
docs = loader.load()

Async LangChain (AsyncHtmlLoader)

If you're running async agents or using AsyncHtmlLoader, the proxy configuration is different. Async components use aiohttp or httpx, not the requests library. Environment variables don't propagate to these clients automatically.

import asyncio
import aiohttp
from langchain_community.document_loaders import AsyncHtmlLoader

async def load_with_proxy(urls: list[str]) -> list:
    proxy_url = "http://username:password:[email protected]:6011"

    # Pass proxy via requests_per_second and header settings
    loader = AsyncHtmlLoader(
        urls,
        proxies={"http://": proxy_url, "https://": proxy_url},
        requests_per_second=2
    )
    return await loader.aload()

docs = asyncio.run(load_with_proxy(["https://example.com"]))

CrewAI Proxy Setup

CrewAI's architecture matters here. An agent crew has agents, tasks, and tools. The proxy configuration needs to cover the tools, because that's where the web requests actually happen. The agents themselves just orchestrate LLM calls; those are handled by LangChain's chat models underneath.

Global Proxy via Environment Variables

Simplest approach. Set env vars before importing CrewAI, and all outbound requests from tools pick them up.

import os

# Must be set before importing crewai
os.environ["HTTP_PROXY"]  = "http://username:password:[email protected]:6011"
os.environ["HTTPS_PROXY"] = "http://username:password:[email protected]:6011"

from crewai import Agent, Task, Crew
from crewai_tools import SerperDevTool, WebsiteSearchTool

search_tool = SerperDevTool()
web_tool    = WebsiteSearchTool()

researcher = Agent(
    role="Market Researcher",
    goal="Gather competitive intelligence from target sites",
    backstory="Expert at finding and extracting market data",
    tools=[search_tool, web_tool],
    verbose=True
)

Per-Agent Proxy Assignment

This is the pattern most guides skip. When you're running a crew with agents that scrape different target types, you want each agent on a different IP. A researcher hitting news sites doesn't need the same proxy subnet as an agent scraping e-commerce product pages. Identity isolation matters at scale.

The cleanest way to do it: wrap your tool calls inside custom tool classes and set the proxy environment variables at call time.

import os
import requests
from crewai_tools import BaseTool
from pydantic import Field

class ProxiedWebFetcher(BaseTool):
    name: str = "Proxied Web Fetcher"
    description: str = "Fetches a URL through a dedicated proxy"
    proxy_url: str = Field(description="Proxy URL for this agent")

    def _run(self, url: str) -> str:
        proxies = {
            "http":  self.proxy_url,
            "https": self.proxy_url,
        }
        response = requests.get(
            url,
            proxies=proxies,
            timeout=30,
            headers={"User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)"}
        )
        response.raise_for_status()
        return response.text[:5000]

# Each agent gets its own proxy
researcher_tool  = ProxiedWebFetcher(proxy_url="http://user:pass:[email protected]:6011")
analyst_tool     = ProxiedWebFetcher(proxy_url="http://user:pass:[email protected]:6011")

CrewAI and SOCKS5: The socksio Requirement

CrewAI uses httpx internally for MCP server connections and some tool requests. If you're routing through a SOCKS5 proxy, httpx needs the socksio package to handle that protocol. Without it, you get a cryptic error.

pip install socksio httpx

import httpx

# Configure httpx client with SOCKS5 proxy
proxy_client = httpx.Client(
    proxy="socks5://username:password:[email protected]:6014",
    timeout=30
)

# Use this client in tools that accept an httpx client parameter
# For env-var-based routing:
import os
os.environ["HTTP_PROXY"]  = "socks5://username:password:[email protected]:6014"
os.environ["HTTPS_PROXY"] = "socks5://username:password:[email protected]:6014"

Claude Agents Proxy Setup

Claude Agents and Claude Code support two distinct proxy methods, and they solve different problems. One routes the API calls themselves. The other routes everything.

ANTHROPIC_BASE_URL: Routing LLM API Calls

This is the cleanest way to put something between your agent and Anthropic's API. Set ANTHROPIC_BASE_URL to point at your proxy, and every sampling request goes through it. Your proxy receives the full plaintext request, can inspect or modify it (injecting credentials, adding logging, rate limiting), then forwards to api.anthropic.com.

# Route all Claude API calls through your proxy
export ANTHROPIC_BASE_URL=http://your-proxy-server:8080
export ANTHROPIC_API_KEY=sk-ant-...

# Claude Code automatically uses ANTHROPIC_BASE_URL
claude

import anthropic

# Custom base URL routes API calls through your proxy
client = anthropic.Anthropic(
    api_key="sk-ant-...",
    base_url="http://your-proxy-server:8080"
)

message = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}]
)

HTTP_PROXY / HTTPS_PROXY: System-Wide Routing

For routing all Claude agent traffic through a proxy, including both API calls and any tool-based web requests, use the standard environment variables.

export HTTP_PROXY=http://username:password:[email protected]:6011
export HTTPS_PROXY=http://username:password:[email protected]:6011

# Claude Code and the Agent SDK respect these variables
claude

import httpx
import anthropic

# Fine-grained control: configure the underlying httpx client directly
http_client = httpx.Client(
    proxy="http://username:password:[email protected]:6011"
)

client = anthropic.Anthropic(
    api_key="sk-ant-...",
    http_client=http_client
)

# All SDK requests now go through the proxy

Choosing the Right Proxy Type for Your Agent's Web Access

Datacenter proxies get blocked. That's the short answer for anyone shipping a production agent that scrapes real sites. Search engines, news sites, e-commerce platforms, any site running Cloudflare Bot Management: they all block datacenter IP ranges either immediately or after a few requests.

For most agent tool requests, rotating residential proxies handle the job. Your agent's WebBaseLoader and search tools see clean IPs on each request, and the target sites treat them as regular consumer traffic.

One thing worth flagging: rotating proxies and authenticated sessions don't mix. If your agent needs to log in and maintain a session across multiple requests, set sticky sessions on the proxy. TorchProxies supports sticky sessions configurable from the dashboard. Without this, each request gets a new IP, the session breaks, and your agent spends half its time re-authenticating.

Production Patterns Worth Knowing

Retry Logic With Backoff

Agents hitting protected sites will get 429s and 403s despite clean proxies. Sometimes it's a rate limit. Sometimes the IP rotation hasn't kicked in yet. Your agent needs to handle these gracefully rather than failing the entire task.

import time
import requests
from requests.exceptions import ProxyError, ConnectionError

def fetch_with_retry(url: str, proxies: dict, max_retries: int = 3) -> str:
    wait = 2
    for attempt in range(max_retries):
        try:
            response = requests.get(
                url,
                proxies=proxies,
                timeout=30,
                headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
            )
            if response.status_code == 429:
                time.sleep(wait)
                wait *= 2  # Exponential backoff
                continue
            response.raise_for_status()
            return response.text

        except (ProxyError, ConnectionError) as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(wait)
            wait *= 2

    raise Exception(f"Failed to fetch {url} after {max_retries} attempts")

Testing Your Proxy Before Running the Agent

Spending 30 seconds on this before a long-running agent job is worth it. Proxy failures mid-run are expensive, especially if you're paying per token for the LLM calls the agent is already making.

import requests

def verify_proxy(proxy_url: str) -> bool:
    proxies = {"http": proxy_url, "https": proxy_url}
    try:
        r = requests.get(
            "https://httpbin.org/ip",
            proxies=proxies,
            timeout=10
        )
        data = r.json()
        print(f"Proxy IP: {data['origin']}")
        return True
    except Exception as e:
        print(f"Proxy check failed: {e}")
        return False

# Run this before your agent starts
proxy_url = "http://username:password:[email protected]:6011"
if not verify_proxy(proxy_url):
    raise RuntimeError("Proxy unreachable. Check credentials and endpoint.")

A Note on .env Files

If you're loading proxy credentials from a .env file via python-dotenv, make sure load_dotenv() runs before any LangChain or CrewAI imports. The same import-time caching issue applies. Structure your entry point like this:

# 1. Load env FIRST
from dotenv import load_dotenv
load_dotenv()

# 2. Then import everything else
from langchain_community.document_loaders import WebBaseLoader
from crewai import Agent, Task, Crew
import anthropic

This is worth getting right before you scale. A subtle import order bug doesn't surface in development on your own IP. It surfaces at 2am when a production agent is halfway through a crawl job.

TorchProxies for AI Agents

From the dashboard, generate credentials and select the proxy type that matches your agent's workload. Plan X proxies use pool-specific hosts and ports — the global residential pool (geox) runs on port 6011, and the SOCKS5 endpoint (socksgeox) runs on port 6014. Targeting parameters (country, city, session) are appended to the credentials in the proxy string.

For research agents hitting search engines and news sites, Plan X at $5/GB is the right starting point. The hybrid infrastructure combines over 50 million real residential IPs with dedicated ISP proxy pools, giving you a choice between authentic residential traffic for maximum trust and unlock rates, or ISP proxies for speed-sensitive operations. Country, city, and state targeting are set directly in the proxy string by appending parameters after the password — no separate dashboard configuration needed.

For authenticated agent workflows where the same account session needs to persist across multiple requests, ISP Static proxies at $2.3/IP/month give you a fixed dedicated IP that doesn't rotate. For Plan X sticky sessions, append a session-XXXXXXXX parameter directly to the proxy string (exactly 8 lowercase alphanumeric characters). Sessions persist for up to 168 hours.

No rate limits on any plan. Free trial available without a credit card. Pay-as-you-go on bandwidth, so you're only paying for what your agents actually use.

import os
from dotenv import load_dotenv

load_dotenv()

# TorchProxies credentials from env
TORCH_USER = os.environ["TORCHPROXIES_USER"]
TORCH_PASS = os.environ["TORCHPROXIES_PASS"]
# Plan X: pool-specific host + port; targeting params after password
PLAN_X_HOST = "geox.torchproxies.com"  # global residential pool
PLAN_X_PORT = "6011"
COUNTRY     = "country-us"  # use city-boston or session-a1b2c3d4 for additional targeting

os.environ["HTTP_PROXY"]  = f"http://{TORCH_USER}:{TORCH_PASS}:{COUNTRY}@{PLAN_X_HOST}:{PLAN_X_PORT}"
os.environ["HTTPS_PROXY"] = f"http://{TORCH_USER}:{TORCH_PASS}:{COUNTRY}@{PLAN_X_HOST}:{PLAN_X_PORT}"

from langchain_community.tools import DuckDuckGoSearchRun
from langchain_community.document_loaders import WebBaseLoader
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_react_agent
from langchain import hub

llm        = ChatOpenAI(model="gpt-4o", temperature=0)
search     = DuckDuckGoSearchRun()
tools      = [search]
prompt     = hub.pull("hwchase17/react")
agent      = create_react_agent(llm, tools, prompt)
executor   = AgentExecutor(agent=agent, tools=tools, verbose=True)

result = executor.invoke({"input": "What are the top AI companies in 2026?"})
print(result["output"])

Quick Reference

Frequently Asked Questions