20-question benchmark
Data used ↗
Whole dataset: sfd-anonymous/sfd-v1 ↗
Download only the 2025 parsed Parquet files:
The command downloads 2025 source files. Each benchmark's recorded source/index snapshot is documented on its page.01 / Retrieval strategy comparison
02 / Correctness vs. cost
03 / Quality by retrieval strategy
Mean correctness /100, pooled across both databases. Each model bar represents 40 answers to the same 20 questions.
Bars start at 60. Use the score-range control for the full 0–100 view.
Experiment scope and metric definitions
The index contains the initial 1,000 source records plus 52 added source records. Seven records repeat an existing document ID, leaving 1,045 unique indexed IDs. Both databases used this same index. Agentic Search also had access to the six source Parquet files. Verified index counts and queries ↗
Correctness is the recorded Haiku score out of 100. Cost uses recorded USD and the selected cost basis; unknown charges remain unknown. Means describe this 20-question sample.
04 / All 16 model, database and strategy conditions
05 / Question-by-question scores
Inspect the evidence behind a score
Read the question, reference answer, model answer, and judge explanation.
Reference answer
Model answer
Source evidence and agent tool details
The experiment
Reproducible selection
Model configuration
What the cost includes
Checks before the run
Resources and concurrency
Completion and recovery
Limits of this result
- Twenty sampled questions are a small evaluation set. Differences are descriptive, not proof of model superiority.
- The random sample has no extreme or unanswerable questions; it cannot estimate performance in those categories.
- Judge scores are model assessments. Review numerical answers and partial credit using the inspector.
- Filing recall measures accessions, not whether every required fact was retrieved. Missing or failed answers are shown explicitly.