# Read-only audit: all 100 questions

Cases scored: **399/400**. Known reported API spend: **$7.950589285**; 4 unpriced attempts.

HTTP attempts: 3441 total, 3437 successful, 4 failed. No model calls were made by this audit.

## ERROR: 399/400 cases have final answers and Haiku scores

Q024 / Gemini Flash / sql_streaming returned a tool call during forced finalization and was not judged.

## WARNING: 4 failed HTTP attempts are retained

Failed HTTP attempts were followed by later successful replies for their tags. This differs from the terminal case-level failure.

## WARNING: Billing contains unpriced failed attempts

Known reported charges are a subtotal. Missing usage on HTTP errors is unknown, not confirmed zero.

## WARNING: 311 tool error responses

SQL schema/syntax errors, duplicate filing-ID rejection and bounded allocation failures can consume the eight-tool budget and cause abstention. A scored case is not proof of successful retrieval.

## WARNING: 176333 duplicated filing-ID groups inside source files

Scoped SQL and filing-read tools reject duplicate IDs. Next experiments should distinguish identical stored-checksum copies from conflicting records and support an explicit row identity. No originals were deduplicated or altered.

## WARNING: Memory reached 2048 MiB; 0 OOM kills

The process-tree limit prevented machine-wide growth. Reaching memory.max still caused pressure and some worker allocation failures; no swap was used.

## INFO: Scores are judge estimates over scored cases

The primary mean excludes the failed case. A separate coverage-adjusted mean treats missing scores as zero without changing the original score. Citation support and correctness are distinct; neither is a guarantee of objective truth.

## INFO: Shared-cache, concurrent-run latency

Four concurrent cases, one serial data tool and a shared cache mean queueing and cache order affect timing. This is not an independent cold-cache latency comparison. The same 100 questions repeat across four conditions, so do not treat 400 scores as independent questions.

## INFO: Reference and evidence limitations

Judge inputs and scores were mechanically checked against raw payloads. Gold quotes were checked for visibility in retrieved outputs, not reverified against every full source filing. Expected-filing recall only counts explicit accession IDs in structured outputs. Long-excerpt flags are heuristics, not proof of leakage.

## WARNING: Some questions already supply the financial operands

Q053, Q059, Q076, Q083 and Q085 include numbers sufficient for substantial arithmetic answers. High correctness with little retrieved text can therefore reflect calculation from the question, rather than successful document retrieval. For example, Q059 / Flash / SQL used a constant arithmetic SELECT; Haiku awarded 98 correctness and 80 citation support. Report correctness and retrieval indicators separately.

## PASS: Saved payloads, scores and summaries reconcile

See structured audit checks and the original-preservation manifest. No requests were retried and no answers were rejudged.

## Scores as recorded

| Model | Strategy | Scored | Mean correctness | Missing=0 mean | Known USD |
|---|---|---:|---:|---:|---:|
| google/gemini-3.8-flash | sql | 100/100 | 66.63 | 66.63 | 3.260522 |
| google/gemini-3.8-flash | sql_streaming | 99/100 | 71.79 | 71.07 | 4.017495 |
| openai/gpt-6-luna | sql | 100/100 | 65.58 | 65.58 | 0.300701 |
| openai/gpt-6-luna | sql_streaming | 100/100 | 55.73 | 55.73 | 0.371705 |

## Next experiments

- Keep this run unchanged. Recover the missing answer/judgment only in a separately recorded recovery experiment, if desired.
- Handle duplicated filing IDs with exact-row selection or a verified identical-content rule; do not blindly choose conflicting copies.
- Strengthen final-answer enforcement and record exhausted-budget requests separately from valid answers.
- Give models clearer schema repair feedback; account for worker allocation failures and consider streaming before text SQL.
- Use uncached or independently warmed runs for latency comparisons. Validate a blinded sample of Haiku judgments and gold references manually.
- Reconcile unpriced HTTP errors with gateway billing; do not replace unknown charges with zero.

Full details: `audit.json`, `corpus-audit.json`, `benchmark.json`; the HTML dashboard uses a compact export and lazy case files.