How to Extract Structured JSON from Any Website (Without Writing Selectors)

You want the price, title and stock status from a product page as JSON. Not HTML, not markdown, but an object your code can use. There are three realistic ways to get there, and which one fits depends on how many sites you scrape and how often they change.

Option 1: CSS selectors

The classic approach: fetch the HTML, parse it, and pick out elements.

import requests
from bs4 import BeautifulSoup

html = requests.get("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html").text
soup = BeautifulSoup(html, "html.parser")

product = {
    "title": soup.select_one("h1").text,
    "price": float(soup.select_one(".price_color").text.lstrip("£")),
    "in_stock": "In stock" in soup.select_one(".availability").text,
}

Good: fast, free, deterministic. Bad: every site needs its own selectors, and they break silently when the markup changes. Cleaning (currency symbols, whitespace, "In stock (22 available)") is your job.

Use it when you scrape a few sites you control or monitor closely.

Option 2: Check for embedded structured data first

Many e-commerce, recipe, job and article pages already ship machine-readable data as JSON-LD for search engines. Before you write a single selector, look for it:

import json
from bs4 import BeautifulSoup

for tag in soup.select('script[type="application/ld+json"]'):
    data = json.loads(tag.string)
    if data.get("@type") == "Product":
        print(data["name"], data["offers"]["price"])

Good: clean, typed and stable, because sites keep it accurate for Google. Bad: it's only there on some pages, often incomplete, and the shapes vary between sites.

Option 3: LLM extraction with a schema

Describe the fields you want and let a language model read the page. Modern APIs can constrain the output to a JSON Schema, so the response always parses and always has the right types: no more "the model wrapped it in markdown".

The steps:

1. Fetch the page and strip it down to readable text (drop scripts, styles and navigation). This cuts tokens, and therefore cost, by 5–10x.
2. Send the text plus your schema to the model, with an instruction to return null instead of guessing.
3. Validate and store.

With our extraction API that's a single request:

curl https://bestaiscraper.com/api/v1/extract \
  -H "Authorization: Bearer $BAS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "fields": {
      "title": "string",
      "price": "number — price in GBP",
      "in_stock": "boolean",
      "rating": "number — star rating from 1 to 5"
    }
  }'

{ "title": "A Light in the Attic", "price": 51.77, "in_stock": true, "rating": 3 }

The same field list works on a different bookshop with completely different markup. That's the real win.

Good: one schema works across many sites, survives redesigns, and can normalise or summarise as it extracts (the rating above is shown as star icons on the page, not as a number). Bad: costs money per page, is slower than a selector, and needs guardrails against hallucinated values. Always ask for null when a value is missing, and spot-check results.

Which should you use?

For that last case, a fetch-only approach won't get past the blocking. Compare tools that handle it in our guide to the best web scraping APIs: Firecrawl and ScrapingBee both combine rendering with built-in AI extraction.

Tips for reliable LLM extraction

• Name fields precisely. price is ambiguous on a page with a sale price and an RRP. Write price: number — current selling price, not the crossed-out price.
• Use list mode for listings. Ask for an array of items rather than calling once per item.
• Keep schemas flat. Deeply nested schemas cost more tokens and fail more often.
• Treat page text as untrusted. Pages can contain text like "ignore previous instructions". Keep your instructions in the system prompt and wrap the page content as data.
• Log token usage. Long pages get expensive, so truncate or pre-filter boilerplate.

You can try all of this without signing up in the free playground.