Product card data by article number

The service fetches product card data by article number and serves it over HTTP: access keys, plans, daily quotas. On top of that, competitor price tracking: “mine and theirs” groups, price history, a card quality score. The platforms offer no partner interfaces, so the data comes from where the site itself takes it.

The card is split in two — by source availability, not for looks

The unchanging part — name, brand, description, specs — sits in an open distribution and is barely throttled. Price, stock and rating come from a protected channel that needs a clean address. Each part has its own freshness: a day against five minutes.

The obvious approach is to hit one main product endpoint. That is also the most heavily protected one: from a server address it refuses, and the card is not returned at all — even though four fifths of the fields live where access is allowed. The split keeps the reliable half always available. A single freshness would not do either: descriptions do not change for months, prices change hourly.

The address of the static part is computed from the article number, not looked up

Host and path are derived arithmetically from the product number — with no preliminary request to the platform.

The obvious route is to open the product page and pull the link out. That is an extra request to exactly the layer that bans: the page goes through protection, the distribution does not. By computing the address locally the protected perimeter is not touched for the static part at all. The price: the range table goes stale, so values outside it fall back to the next host — the card is served instead of a hard refusal.

The response always states what happened to each part

Alongside the data comes a per-channel note: fresh, cache, not found, refused, skipped.

The key case is a product that exists, whose static part arrived while the protected channel answered “no such thing”. A naive parser writes “product not found” — a quiet lie: the client sees an empty price and concludes the article does not exist. Analytics are built on this data: silently turning “blocked” into “no price” would shift every average and median.

The address pool raises “exhausted” instead of an empty result

Exit points are handed out in rotation; a blocked one goes into cooldown. When none are free, an error is raised rather than emptiness returned.

The obvious implementation is “no address, go direct” or “return nothing”. The first burns the machine’s own address exactly when the platform is banning hardest. The second is indistinguishable from “the product does not exist”: the client gets a successful response with empty fields and records that in the price history as fact. An honest error stops the load and allows a retry later.

Cooldown is a timestamp, not a failure counter

Each address stores the moment until which it must not be touched; availability is a comparison with the current time.

A retry counter would have to be reset on a timer — that is, a background task for the sake of one variable. A timestamp recovers on its own at the next request. And one detail: a successful request resets the error counter but does not lift the cooldown — a recently banned address should not be loaded up again because of one success.

The cache is shared across clients

The cache key is platform and article number, with no client in it. A request for a hundred articles takes everything still fresh from the cache and goes out only for the rest.

A per-client cache feels more natural but breaks the economics: one popular product is asked for by dozens of clients, and every request costs an address and a ban risk. Card data is public and identical for everyone — there is nothing there to separate.

The per-minute limit lives in memory, the daily quota on disk

Rate limiting is a sliding window in memory. The daily quota is a counter in the database, updated atomically.

The asymmetry is deliberate. Losing the per-minute limit on restart costs nothing, while writing to the database on every request is needless work on the hot path. The daily quota is billing: resetting it on restart means free requests as many times as there were restarts.

Tracking separates “who watches what” from “what happened to the product”

One table holds a client’s subscriptions, another holds price snapshots per product with no client attached. The card quality score is computed at snapshot time and stored with it.

Price history is objective; it belongs to nobody. Storing it per client means that for two clients watching one competitor you keep two histories and make two requests. And the score is written immediately because otherwise a “how card quality changed” chart would require re-fetching a product that may no longer be on sale.

The price recommendation compares against competitors, not the whole market

Your own products are excluded from the comparison base; the average and the bounds are computed from others only.

The obvious implementation is an average across the whole niche. Then your own price enters the base it is compared against: for a seller with three cards the verdict is half built on their own prices — raise the price, the average follows, the service says “fine”. The edge case is caught separately: no competitors with a price is not “you are in the market” but “nothing to compare to”.

Python · FastAPI · SQLite · systemd