100-question benchmark on 6 Parquet files (~432K document IDs)
Data used ↗
Whole dataset: sfd-anonymous/sfd-v1 ↗
Download only the 2025 parsed Parquet files:
The command downloads 2025 source files. Each benchmark's recorded source/index snapshot is documented on its page.In data/parsed · 431,952 unique document IDs · 6 Parquet files · 6.44 GiB · 638,067 stored records (includes repeated IDs)
01 / Model comparison Agentic Search · two tool sets
- SQL: The agent inspects schemas and runs read-only DuckDB queries over Parquet to find evidence.
- SQL + streaming: Adds company lookup and paginated filing reads to retrieve source passages with offsets.
Cost / correctness
All strategies
The experiment
Verified counts in these six files
Counted directly in data/parsed with DuckDB SQL and an independent PyArrow scan. Document IDs use file_stem; company IDs use accession. Repeated IDs count once. These are local source-file counts.
| Source file | Records | Document IDs | Company IDs |
|---|
A separate scan read the first 5,000 characters of every text row: 638,067 rows and 431,952 unique accession IDs inside the text, agreeing with the metadata counts. Complete row/header and filing-type verification ↗
How to read the scores and costs
Avg score overall and difficulty scores are mean Haiku correctness /100 over scored cases; hover over a difficulty score for its scored-case count. One case is unscored. Cost shows the known answer + judge total (two decimals) / average per benchmark request (six decimals to show small costs). Tool calls show total / average per benchmark request. A request here is one question for one model and strategy, including failed cases. Unpriced attempts are noted on hover and with hollow chart points, so final billing may be higher. Avg time per question includes queueing for each strategy. The chart uses logarithmic known cost per planned case on X and mean correctness on Y. Hover over a strategy to highlight it; click to isolate it.
Sources and original results are read-only. No answers were retried or rejudged for this dashboard.
Read the audit report ↗Avg score by question difficulty
Models and providers
| Role | Model | Pinned provider | Reasoning | Temperature |
|---|
Provider fallback disabled. Temperature is omitted where unsupported. All costs are API-reported USD; unknown charges remain unknown.
All 100 questions
One row per question, with a score for each condition. Click any score or “Inspect” for the exact answer, retrieval steps, sent context and judge rationale.
Answers and agent activity
400 planned cases. A status of “scored” means a final answer and valid judge JSON exist; it does not mean the answer is correct. Costs include all associated API attempts. Use Columns to show or hide answers, references, rationale, tokens and other metrics.
What passed, what needs attention
Tool errors
Failed HTTP attempts
The audit validates recorded payloads, score parsing, counts, providers, costs and source metadata. It is not a new factual adjudication of every answer or a full revalidation of every gold quote. Some questions supply calculation inputs directly.
Files and reproducibility
Rebuild with agentic-search-only/scripts/analyze-results.sh from the repository root. The analyzer reads the run and writes only to analysis/all100/. It never invokes a model.