04 / REASONING LEVELS
Compare reasoning levels.
One point = one model at one reasoning level, averaged across the image set. Compare Low, Medium and High.
Mean error is the average absolute percentage count deviation. Lower is better. All values come from saved JSON.
Reading the saved eval4 results…
Data coverage and comparison limits
Cost vs Mean Error
Compare the total cost and mean error across all images. Left = cheaper. Up = less error. Zero error is at the top. The line connects Low → Medium → High for each model.
Timings measured on Colab; provider presets estimate rental cost using those same timings. They include the GPU and host CPU/RAM, exclude extra storage, network and taxes, and are dated reference rates. Only DL cost changes; LLM charges and counting error stay fixed.
▲ Colab DL · one CUDA inference configuration per model · select a compute rate above to estimate cost.
Hover, tap or keyboard-focus a point for its cost and counting deviation.
Counting quality: LLMs and Colab DL models
Mean absolute percentage count deviation on the same image set. Lower is better. No hourly rate is needed to compare counts.
DL models use text-only settings. The selected FSC-147 images include training examples for dataset-trained checkpoints. Colab compute estimates price inference only, excluding setup, warm-up and idle time. DL detection scores are not count confidence.
Mean Confidence vs Mean Count Accuracy
Each point averages the same six images for one LLM and reasoning level. Both axes run from 0–100%. On the dashed diagonal, confidence matches mean count accuracy. Bottom-right = overconfident. Top-left = underconfident.
Select a model-level point to see its mean confidence and mean count accuracy.
Optional: Error vs Coverage — does trusting confidence help?
Error vs Coverage
Each curve is one model and reasoning level. Moving right keeps more answers. If the curve rises as less confident answers enter, error increases: the highest-confidence subset counted more closely.
Select a curve point to see its confidence cutoff, accepted images and mean error.
Questions & answers
Answers update with your filters. These are observations from a small pilot.
Detailed results & raw JSON
Every model, every level
Partial runs remain visible here. Full-image comparisons require complete predictions.
| Model / reasoning | Answers | Mean error | Cost, USD | Mean confidence | Token cap | Source |
|---|
Methods, limitations and source files
Mean counting error = mean of |prediction − supplied count| ÷ supplied count × 100. Every scored image has equal weight. Zero or conflicting supplied counts cannot be used for percentage deviation. Predictions are never rounded before scoring.
LLM cost is the saved API charge. Colab DL compute cost is total inference seconds ÷ 3,600 × the selected hourly rate. Saved rates come from the benchmark JSON; provider reference rates and source links come from dl-pricing.json. Custom and provider selections reprice the view without editing results. Missing rates stay unknown; installation, loading, warm-up and idle time are excluded. Imported DL results appear in the cost and optional quality comparisons and remain excluded from confidence charts. Cost points require complete predictions and known charges. Confidence points require valid counts and confidence for the same full image set. Partial runs remain in the table and may have coverage curves ending below 100%.
Thin cost connectors and low-to-high conclusions require matching recorded settings within a model, apart from effort. Missing intermediate levels break the connector. Different token caps, provider endpoints and sampling settings across models limit causal comparisons. Settings are exposed in the table and source files.
Confidence remains P(exact count). Mean count accuracy averages the per-image min/max count ratio, with both zero defined as 100% and only one zero as 0%. Comparing this count closeness with mean confidence is not formal probability calibration. Coverage curves use confidence-ranked subsets within each model/effort configuration. Pooled cutoff summaries repeat the same images across configurations; they are not independent new image samples. With this small, selected pilot, useful cutoffs need further evaluation.
Predictions and confidence: eval4/ only; smoke tests and other evaluation folders are excluded. Supplied counts: existing scoring metadata, joined by group and image. These counts have not been independently audited here. Read the runner. No model API requests are made by this page.
Dataset: FSC-147 / Learning To Count Everything, Viresh Ranjan, Udbhav Sharma, Thu Nguyen and Minh Hoai (CVPR 2021). Source images · Paper · Recorded download URL. This page covers selected Group 1 images, not the full FSC-147 benchmark.