How to Scrape a Website Without Getting Blocked (2026 Guide)

If you want to scrape a website without getting blocked, start by working out how you are being blocked. A 403, a 429, a CAPTCHA page and a "200 OK" with an empty product list are four different problems with four different fixes. Throwing proxies at all of them wastes money.

This guide goes in order of cost: fix the cheap, common causes first, and only reach for browsers and residential proxies when the evidence says you need them.

Step 0: Read the block

Log the status code, the response size and the first 200 characters of the body for every failed request. Then match it against this table.

Check whether the page is a challenge before you parse it, so a blocked response never ends up in your database as "no results":

def looks_blocked(r) -> bool:
    body = r.text[:5000].lower()
    return (
        r.status_code in (403, 429, 503)
        or "captcha" in body
        or "verify you are human" in body
        or len(r.text) < 2000
    )

Step 1: Check what you are allowed to do

Read robots.txt and the site's terms before you build anything. Scraping public pages is common, but terms of service, login walls and personal data change the picture, and what is legal depends on your jurisdiction and the data involved. If the site offers an API or a data feed, use it: it is faster, stable and never blocked. We are not lawyers, so get advice before scraping anything personal or behind a login.

Step 2: Send requests like a browser

The cheapest blocks are triggered by default client behaviour. requests announces itself as python-requests/2.x, which many sites reject outright.

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
})

r = session.get("https://example.com/products")

Three details matter more than people expect:

• Use a Session. It keeps cookies, which real browsers do. A client that discards every cookie looks odd.
• Keep headers consistent. A Chrome User-Agent with Firefox-style Accept headers is a stronger bot signal than a plain honest one.
• Update the version. A User-Agent claiming a browser from several years ago is easy to flag. Copy a current one from your own browser's dev tools.

Step 3: Slow down, and add jitter

Most 429s are self-inflicted. Fixed intervals look robotic too, so randomise them and back off when the server tells you to:

import random, time

def polite_get(session, url, max_tries=5):
    for attempt in range(max_tries):
        r = session.get(url, timeout=30)
        if r.status_code not in (429, 503):
            return r
        wait = int(r.headers.get("Retry-After", 2 ** attempt * 5))
        time.sleep(wait + random.uniform(0, 3))
    raise RuntimeError(f"still rate limited: {url}")

for url in urls:
    page = polite_get(session, url)
    time.sleep(random.uniform(1.5, 4.0))

If you need volume, run several slow workers on different IPs rather than one fast worker on one. Concurrency per domain is the number to keep low, not total throughput.

Step 4: Look for the JSON the page itself uses

Many "JavaScript-heavy" pages are not hard to scrape at all. The page calls an internal JSON endpoint, and you can call it directly: faster, cheaper and less likely to be challenged than loading a full browser.

1. Open dev tools, go to Network, filter by Fetch/XHR, and reload.
2. Find the request that returns the data you see on screen.
3. Right-click it, Copy as cURL, and reproduce it in your code.

Also search the page source for __NEXT_DATA__ or application/ld+json. Plenty of sites embed the full data as JSON in the initial HTML, which needs no rendering at all. Our guide to extracting structured JSON covers this in detail.

Be aware that internal endpoints can carry tokens that expire, and they change without notice. Treat them as a faster path, not a contract.

Step 5: Use a real browser when the page needs one

If the data only appears after JavaScript runs, or the site serves a challenge that a plain HTTP client cannot pass, render the page in a headless browser.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(locale="en-US", viewport={"width": 1366, "height": 768})
    page.goto("https://example.com/products", wait_until="networkidle")
    page.wait_for_selector(".product-card")
    html = page.content()
    browser.close()

Browsers cost far more per page than HTTP requests, in CPU, memory and time. Block what you do not need and you recover a lot of it:

page.route("**/*", lambda route: route.abort()
           if route.request.resource_type in ("image", "font", "media")
           else route.continue_())

Know the limits. A default headless Chromium leaks automation signals, and protected sites check them along with the TLS handshake, which differs between a real browser and most HTTP libraries. Stealth patches exist, but they are an arms race you will keep losing time to. If a site invests in serious bot protection, that is the point to stop maintaining your own evasion and pay for someone whose full-time job it is.

Step 6: Fix your IP reputation

When the block depends on where the request comes from, headers will not help. Traffic from cloud provider ranges is treated with suspicion by many sites, and a single IP sending thousands of requests will be limited no matter how human it looks.

Rotate per request for stateless pages, and keep a sticky session (the same IP for several requests) whenever you log in, paginate with a cursor or fill a cart. Rotating mid-session is a common way to trigger a block.

Use proxies from providers that source their networks with consent. Providers such as Bright Data and Oxylabs sell proxy networks, and each tool page shows what we verified from the vendor's own site.

Step 7: Decide whether to build or buy

Maintaining all of the above is real engineering work. The question is whether that work is your product.

A scraping API charges you per successful request and handles rendering, IP rotation and challenge handling behind one endpoint. Compare them side by side in our web scraping API rankings, or see Firecrawl and ScrapingBee for two that also offer AI extraction.

If you only need the data and not the HTML, skip the parsing layer entirely: our extraction API takes a URL and a field list and returns validated JSON. You can try it in the free playground.

A checklist before you scale up

• I read robots.txt and the terms, and I am not collecting personal data I should not.
• Every response is checked for block pages, not just status codes.
• Requests use a session, consistent headers and randomised delays.
• I looked for a JSON endpoint or embedded data before reaching for a browser.
• Failures are retried with exponential backoff and a hard cap.
• Per-domain concurrency is low and configurable.
• I track the success rate per domain, so a silent block shows up as a chart, not a surprise.

The last one matters most. Blocking is rarely a one-off: sites change their defences, and the scraper that worked in March fails in June. Measure the success rate, and you will know the day it happens.