How scraping-bench grades
Every task is pass or fail, and a script decides it. Claude Haiku 5.5 only does the task; no model grades anything.
One task
- The model gets the task, one tool and the same prompt for every provider. The tool is
scrape(url),search(query),map(site)orcrawl(site, scope), and the provider behind it is the only thing that changes. - It may call the tool a fixed number of times, within a time and dollar limit per task: scrape 2 calls, 60 s, $0.10; search 3 calls, 60 s, $0.10; map 2 calls, 2 min, $0.25; crawl 1 call, 10 min, $2.00.
- Long responses arrive in pages of 32 KB, and the model can read every page, so a long response costs more tokens but never hides the answer.
- It answers in JSON with the named fields the task asks for.
- A script compares each field with the answer key. The task passes only if every field matches, the run stayed inside its limits and nothing errored.
What counts as a match formatting is forgiven; values are not
| Field | Rule |
|---|---|
| Text | Case, punctuation and spacing are ignored; the words must equal an accepted answer |
| Number | $168.28 and 168.28 USD parse the same; must fall within the key's tolerance |
| Version | v0.143.3 equals 0.143.3 |
| Date | Oct 17, 2026 equals 2026-10-17 |
| Missing answer | Some pages leave the answer out on purpose; the right reply is null or empty |
| URL list (map) | URLs are normalised (www., trailing slash, tracking parameters). Our test site needs every page and nothing extra; real docs sites need 90% recall and 90% precision |
| Pages (crawl) | The right page URLs, with 90% recall and 90% precision. Page titles are reported separately |
The rules are in src/grade/index.ts, with tests beside it.
Where the answer keys come from
- Scrape. Independent documentation sites, page-checked PyPI projects and four browser-verified JS fixture pages in the full set.
- Search. Official registry APIs, SEC XBRL, Wikidata, GitHub releases, documentation HTML and verified absent npm releases.
- Map. Independent documentation sites; each scope has exact sitemap agreement with a breadth-first HTML crawl.
- Crawl. Independent documentation sites; each scope has exact sitemap agreement with a breadth-first HTML crawl.
Controls
- Model alone: no tool. Shows what the model already knows.
- Correct page given: the right page, fetched independently, is handed to the model. The ceiling for scrape.
Numbers
- Tasks done: the share of tasks that passed.
- Cost per finished task: all API and model spend, failures included, divided by tasks done. Model cost is a list-price estimate.
- Time: the median over tasks that passed.
- Answer in the provider's response: no model. Does the API's own output contain every answer?
- Page coverage: for scrape, the share of 8 sentences sampled from each page's own text that appear in the provider's response.
When a gap counts
Tasks from the same site are not independent, so each provider is compared with its track's leader on the same tasks, with a 95% interval from a bootstrap that resamples whole sites 10,000 times. A gap counts only when its interval excludes zero. 9 gaps do on this run; each track's page lists them.
What was left out
Not tested in v2: Exa, Tavily, Bright Data and Perplexity and Parallel. Jina search left out: it has no hard cap on tokens per call.
Limits
One client location. The model and its settings are part of what is measured, so these are not pure vendor scores. The rules every v2 number is measured by are in MEASURE-v2.md. The full list is in FINDINGS.md. Every number on this site comes from results.json, run wb-v2.