Ecommerce Price Scraping: Build a Price Monitor in Python
November 14, 2026
Ecommerce price scraping sounds simple: fetch a product page, read the price, repeat every day. The hard part is not the fetch. It is getting a correct number out of thousands of differently built pages and noticing when your scraper silently starts returning the wrong one.
This guide builds a small price monitor in Python that reads each page the cheapest reliable way, falls back to LLM extraction when that fails, and stores a history you can alert on. It uses only the standard library plus requests and beautifulsoup4.
The design: cheapest reliable method first
Product pages give you three sources for a price, in order of how much you can trust them:
- JSON-LD structured data. Many shops embed a
Productwith anOfferfor search engines. Google's product structured data docs listpriceas a required property of an offer, withpriceCurrencyas the currency in ISO 4217 format andavailabilityas a recommended one. Because shops want rich results, this data is usually kept in step with the visible price. - LLM extraction from the page text. When a page has no usable JSON-LD, describe the fields you want and let a model read the page. This costs more per page than parsing JSON, so use it as the fallback, not the default.
- CSS selectors per shop. Fastest and free, but each shop needs its own selectors and they break silently on a redesign. Worth it only for the few shops you monitor most. See the three extraction approaches compared.
The script below implements 1 and 2 and records which one produced each row, so you can see how often the fallback fires.
Reading the price from JSON-LD
Two details trip up naive parsers. JSON-LD often wraps nodes in an @graph list, and offers can be a single object or a list. The price can also sit inside a nested priceSpecification instead of directly on the offer, which Google's docs allow. Availability is a schema.org value such as https://schema.org/InStock or https://schema.org/OutOfStock, and Google also accepts the short names without the URL prefix, so compare on the ending.
The full script
import json
import os
import sqlite3
import time
import requests
from bs4 import BeautifulSoup
URLS = [
"https://example-shop.com/products/espresso-grinder",
]
DB = sqlite3.connect("prices.db")
DB.execute(
"CREATE TABLE IF NOT EXISTS prices ("
"url TEXT, ts INTEGER, price REAL, currency TEXT, in_stock INTEGER, source TEXT)"
)
def offers_from_jsonld(html):
"""Return (price, currency, in_stock) from the first Product offer, else None."""
soup = BeautifulSoup(html, "html.parser")
for tag in soup.select('script[type="application/ld+json"]'):
try:
data = json.loads(tag.string or "")
except json.JSONDecodeError:
continue
nodes = data if isinstance(data, list) else data.get("@graph", [data])
for node in nodes:
types = node.get("@type")
if "Product" not in (types if isinstance(types, list) else [types]):
continue
offers = node.get("offers")
offer = offers[0] if isinstance(offers, list) and offers else offers
if not isinstance(offer, dict):
continue
spec = offer.get("priceSpecification") or {}
if isinstance(spec, list):
spec = spec[0] if spec else {}
price = offer.get("price", spec.get("price"))
if price in (None, ""):
continue
currency = offer.get("priceCurrency", spec.get("priceCurrency"))
avail = str(offer.get("availability", ""))
return float(price), currency, avail.endswith(("InStock", "LimitedAvailability"))
return None
def offers_from_api(url):
r = requests.post(
"https://bestaiscraper.com/api/v1/extract",
headers={"Authorization": f"Bearer {os.environ['BAS_KEY']}"},
json={
"url": url,
"fields": {
"price": "number — current selling price, without currency symbol",
"currency": "string — ISO 4217 code such as USD or EUR",
"in_stock": "boolean",
},
"instructions": "Use the price shown for the main product, not related items.",
},
timeout=120,
)
r.raise_for_status()
d = r.json()["data"]
return d["price"], d["currency"], d["in_stock"]
for url in URLS:
html = requests.get(url, timeout=30, headers={"User-Agent": "price-monitor/1.0"}).text
found = offers_from_jsonld(html)
source = "json-ld"
if found is None:
found, source = offers_from_api(url), "llm"
price, currency, in_stock = found
if price is None:
continue # keep the gap rather than storing a guess
prev = DB.execute(
"SELECT price FROM prices WHERE url=? ORDER BY ts DESC LIMIT 1", (url,)
).fetchone()
DB.execute(
"INSERT INTO prices VALUES (?,?,?,?,?,?)",
(url, int(time.time()), price, currency, int(bool(in_stock)), source),
)
DB.commit()
if prev and price < prev[0] * 0.95:
print(f"DROP {url}: {prev[0]} -> {price} {currency}")
time.sleep(2) # be polite between requests
Run it once a day from cron or a scheduled GitHub Action. Each run appends one row per URL to prices.db, and prints a line when the new price is more than 5% below the previous one. Change the 0.95 to taste.
Three choices in the script are deliberate:
- It skips instead of guessing. If neither method finds a price, nothing is stored. A gap in the history is honest; a wrong number triggers false alerts.
- It records the source.
json-ldorllmper row lets you spot a shop whose structured data disappeared. - It pauses between requests. Two seconds is a floor. Slow down further for small shops.
Making LLM extraction safe for prices
A price is exactly the field where a plausible wrong answer does damage. A few rules help:
- Ask for a number, not a string.
"number — current selling price, without currency symbol"keeps "$1,299.00 (was $1,499.00)" from reaching your database. - Allow null. A good extractor returns nothing when the price isn't on the page, rather than inventing one. Our extraction API returns
nullfor fields it can't find, and the script above skips those. - Sanity-check against history. Reject a new price that is under, say, 20% or over 300% of the last one, and flag it for review instead of storing it. Wrong picks (a related product, a monthly installment) usually show up as large jumps.
- Expect no JavaScript. Our API does not execute JavaScript, so a price that only appears after client-side rendering needs a rendering scraper first. Firecrawl and ScrapingBee are two options; each tool page lists what we verified from the vendor's own site, and the web scraping API rankings compare the rest.
You can try the field definition on a real product page in the playground before wiring it into a script.
Staying on the right side of the site
- Check
robots.txtand the terms. RFC 9309 says robots.txt rules are not a form of access authorization, so an allowed path is not the same as permission. Many shops restrict automated access in their terms, and that is a separate question from the file. When a retailer offers a feed or an official API, prefer it. - Keep volume low. One request per product per day is enough for most price tracking, and polite traffic is less likely to be blocked.
- Handle blocks as a signal. If you start getting challenge pages, slow down first. For the technical side, see how to scrape without getting blocked.
From monitor to product
Once you have the history, the useful features are small ones: alert on drops, show the lowest price seen, flag items that go back in stock (in_stock flipping from 0 to 1). A single table and a daily job cover all three.
If you would rather not maintain page parsing across dozens of shops, the fallback path in this script is the part our extraction API replaces: one request per URL with the fields above, typed JSON back.
Get new guides and pricing changes
One email when we publish something useful. No spam.