What we found
- Is this the price of this item? (yes/no). 92.3% of 3,795 decisions were right, but that number flatters it: only 9.1% of the snippets were true matches, so answering "no" every time scores 90.9%. It ranks well (AUC 0.86) and its probabilities are fairly honest (expected calibration error 0.045), but at the usual 0.5 cut-off it finds only 22% of the true matches. Use the probability as a score, not a verdict.
- Which pricing unit is this? (one of the taxonomy's pricing units). 65% correct, and over-confident (expected calibration error 0.19). At confidence 0.9 and above it decides 58% of cases at 81% accuracy: better than average, but not the 98% the stated confidence implies.
- Cost and speed. 4,141 decisions (3,795 yes/no, 346 unit) in 3,795 Jev calls cost $0.061; latency per call: median 88 ms, 95th percentile 132 ms. Fetching the pages that plain HTTP could not read, through Glasser, cost $0.183. Total $0.244.
Method
- Pages. Each of the 626 priced line items cites one or more source pages (139 distinct URLs). We fetched each with plain HTTP and kept a dated snapshot (115 succeeded). Pages that were blocked or returned no prices were fetched once more through Glasser, a pay-per-call data API service (page fetches from Serper), with spend capped in code. Snapshots are not republished.
- Snippets. Code splits each page into table rows and sentences that contain a currency amount. For each line item the positive is the snippet whose amount equals the recorded price. Every other price snippet on the same page is a hard negative: it looks like a price but is not this one.
- Drift. 165 line items were excluded because their page still shows prices but not the recorded one (they may have changed, or be written in a form our matcher misses); 71 because their page showed no dollar amounts at all; 44 because no page could be read. That leaves 346 line items with a positive.
- Questions. For every positive and for up to 12 hard negatives per item (3,449 in all) we asked Jev a yes/no question: does this text state the list price of the item, per its unit and commitment. For positives whose unit is in our taxonomy (346) we also asked, in the same call, which taxonomy unit the price uses.
- Scoring. Code computes everything. A yes/no answer is "yes" at P(yes) ≥ 0.5; its confidence is the probability of the answer given. For the unit question the confidence is Jev's own. We then bin by confidence and compare to observed accuracy.
Reliability
A calibrated system sits on the dashed diagonal: when it says 80%, it is right 80% of the time. Dot size shows how many decisions fall in the bin; bins with fewer than 5 decisions are left out.
Coverage against accuracy
If we only act when Jev's confidence reaches a threshold, how many cases does it decide, and how often is it right on those?
| Confidence ≥ | Yes/no: decided | Yes/no: accuracy | Unit: decided | Unit: accuracy |
|---|---|---|---|---|
| 0.5 | 100% | 92.3% | 91% | 67.2% |
| 0.6 | 97% | 93.7% | 81% | 71.5% |
| 0.7 | 92% | 95.1% | 72% | 77.1% |
| 0.8 | 81% | 96.8% | 66% | 80.7% |
| 0.9 | 61% | 97.8% | 58% | 81.4% |
| 0.95 | 37% | 98.2% | 50% | 82.6% |
| 0.99 | 0% | 100.0% | 36% | 81.0% |
At 0.9 the yes/no question decides 61% of cases at 97.8% accuracy, against 90.9% for always answering "no". Most of that gain is in the negatives; on the rare true matches the picture is the recall figure above.
Caveats
- Imbalance. Positives are 9.1% of the sample, which is why accuracy alone misleads. Real pages have far more non-matching prices than we sampled (up to 12 per item), so precision on a full page would be lower.
- Our labels are not perfect. A positive is a snippet containing the recorded number. A snippet can contain it by coincidence, and the true source sentence can be missing when a page splits a price across table cells. Pages we could not match were dropped, not scored, so results describe pages where the price is still present in a simple form.
- The unit question's truth is our taxonomy label, and some units are close neighbours (per host or per APM host; per GB ingested or processed). Some of the misses may be defensible readings.
- One question wording, one model version, one run. We did not tune the wording, and we did not compare against a general-purpose language model; that is future work.
- The pages are a snapshot taken on 2026-10-14, and vendors change them.
What this does not change
Every number on this site is still computed by code from the open data. A model never supplies a price, quantity or total, and the site works fully with every AI feature switched off. No vendor text appears on this page: only our own labels and aggregate results.
Data and code
The line items and the sources they cite are in the open data (JSON and CSV). The aggregate numbers behind these charts are the ones on this page, from a run on 2026-10-14. The evaluation code is not published yet.