---
title: "How to Extract Structured JSON from Any Website (Without Writing Selectors)"
description: "Three ways to turn a web page into clean JSON, from CSS selectors to LLM extraction, with working code and the trade-offs of each approach."
canonical_url: "https://bestaiscraper.com/blog/extract-structured-json-from-any-website"
---

> Index of every page on this site: [llms.txt](https://bestaiscraper.com/llms.txt)

You want the **price, title and stock status** from a product page as JSON. Not HTML, not markdown, but an object your code can use. There are three realistic ways to get there, and which one fits depends on how many sites you scrape and how often they change.

## Option 1: CSS selectors

The classic approach: fetch the HTML, parse it, and pick out elements.

```python
import requests
from bs4 import BeautifulSoup

html = requests.get("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html").text
soup = BeautifulSoup(html, "html.parser")

product = {
    "title": soup.select_one("h1").text,
    "price": float(soup.select_one(".price_color").text.lstrip("£")),
    "in_stock": "In stock" in soup.select_one(".availability").text,
}
```

**Good:** fast, free, deterministic.
**Bad:** every site needs its own selectors, and they break silently when the markup changes. Cleaning (currency symbols, whitespace, "In stock (22 available)") is your job.

Use it when you scrape **a few sites you control or monitor closely**.

## Option 2: Check for embedded structured data first

Many e-commerce, recipe, job and article pages already ship machine-readable data as JSON-LD for search engines. Before you write a single selector, look for it:

```python
import json
from bs4 import BeautifulSoup

for tag in soup.select('script[type="application/ld+json"]'):
    data = json.loads(tag.string)
    if data.get("@type") == "Product":
        print(data["name"], data["offers"]["price"])
```

**Good:** clean, typed and stable, because sites keep it accurate for Google.
**Bad:** it's only there on some pages, often incomplete, and the shapes vary between sites.

## Option 3: LLM extraction with a schema

Describe the fields you want and let a language model read the page. Modern APIs can **constrain the output to a JSON Schema**, so the response always parses and always has the right types: no more "the model wrapped it in markdown".

The steps:

1. Fetch the page and strip it down to readable text (drop scripts, styles and navigation). This cuts tokens, and therefore cost, by 5–10x.
2. Send the text plus your schema to the model, with an instruction to return `null` instead of guessing.
3. Validate and store.

With our [extraction API](https://bestaiscraper.com/extract) that's a single request:

```bash
curl https://bestaiscraper.com/api/v1/extract \
  -H "Authorization: Bearer $BAS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "fields": {
      "title": "string",
      "price": "number — price in GBP",
      "in_stock": "boolean",
      "rating": "number — star rating from 1 to 5"
    }
  }'
```

```json
{ "title": "A Light in the Attic", "price": 51.77, "in_stock": true, "rating": 3 }
```

The same field list works on a different bookshop with completely different markup. That's the real win.

**Good:** one schema works across many sites, survives redesigns, and can normalise or summarise as it extracts (the rating above is shown as star icons on the page, not as a number).
**Bad:** costs money per page, is slower than a selector, and needs guardrails against hallucinated values. Always ask for `null` when a value is missing, and spot-check results.

## Which should you use?

<table>
<thead>
  <tr>
    <th>
      Situation
    </th>
    
    <th>
      Best approach
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      1–3 sites, stable markup, high volume
    </td>
    
    <td>
      CSS selectors
    </td>
  </tr>
  
  <tr>
    <td>
      Product, recipe, job or article pages
    </td>
    
    <td>
      JSON-LD first, then fall back
    </td>
  </tr>
  
  <tr>
    <td>
      Many different sites, or layouts that change
    </td>
    
    <td>
      LLM extraction
    </td>
  </tr>
  
  <tr>
    <td>
      Heavily protected or JavaScript-heavy sites
    </td>
    
    <td>
      A scraping API with rendering and proxies, plus LLM extraction
    </td>
  </tr>
</tbody>
</table>

For that last case, a fetch-only approach won't get past the blocking. Compare tools that handle it in our guide to the [best web scraping APIs](https://bestaiscraper.com/best/web-scraping-apis): [Firecrawl](https://bestaiscraper.com/tools/firecrawl) and [ScrapingBee](https://bestaiscraper.com/tools/scrapingbee) both combine rendering with built-in AI extraction.

## Tips for reliable LLM extraction

- **Name fields precisely.** `price` is ambiguous on a page with a sale price and an RRP. Write `price: number — current selling price, not the crossed-out price`.
- **Use list mode for listings.** Ask for an array of items rather than calling once per item.
- **Keep schemas flat.** Deeply nested schemas cost more tokens and fail more often.
- **Treat page text as untrusted.** Pages can contain text like "ignore previous instructions". Keep your instructions in the system prompt and wrap the page content as data.
- **Log token usage.** Long pages get expensive, so truncate or pre-filter boilerplate.

You can try all of this without signing up in the [free playground](https://bestaiscraper.com/extract).

## Sitemap

See the full [sitemap](https://bestaiscraper.com/sitemap.md) for all pages.