680+ data points across 3 benchmark dimensions (April 2026). Real Claude API calls, token counts from API metadata. Re-verified July 2026 on current models with added statistical rigor and a net-savings axis — see the note below.
Single-Turn A/B
4 models
176W / 10L / 2T across 188 tests
Multi-Turn Pipeline
23/30 wins
10 real coding tasks across 3 models
MCP Aggregator
85% savings
23 tools tested across 4 models
The April benchmark below is retained in full (it is the deepest per-category and multi-turn dataset we have). In July 2026 we re-ran the core measurement on the current model generation with a stricter method. What changed:
Current models
Haiku 4.5, Sonnet 5, Opus 4.8, Fable 5 (adds Fable; retires Opus 4.6/4.7).
New axis: net savings
Bill reduction after the injected context is counted — 20–48% per model. April measured output reduction only.
Added rigor
Per-prompt aggregation (no pseudo-replication), seeded bootstrap 95% CIs, Cohen's d, a pre-registered publication gate, and adversarial verification of findings.
No temperature=0
Current models reject temperature=0; an N-trial paired design replaces the determinism claim. Quality re-scored with the April-parity rubric.
Effort-controlled finding
Run at equal effort on the shared prompts, Haiku is the strongest (73% output / 58% net savings), Sonnet and Opus tie (59% output), and Fable is weakest (40% / 20%). Opus's earlier apparent lead was an effort artifact — it had been tested at a higher effort than the others. The controlled re-run corrected it.
Honest limits: "up to ~90%" is a best-case prompt, not typical; output reduction is not the same as total-bill reduction; single one-off subscription calls with no reuse can cost slightly more (the benefit needs a session). Full record: TESTRESULTSJULY2026 and the claims audit in the repository.
Each prompt was sent to Claude twice — once bare, once with Superstack. Avg Reduction = fewer output tokens (saves money). Quality Δ = response quality change (positive = better). Win Rate = % of prompts where Superstack used fewer tokens.
52.6%
Avg Reduction
+8
Quality
97.9%
Win Rate
89.1%
Best Reduction
46W / 0L / 1T
Record
33.1%
Avg Reduction
+4.4
Quality
89.4%
Win Rate
89.3%
Best Reduction
42W / 4L / 1T
Record
46.9%
Avg Reduction
+6.4
Quality
93.6%
Win Rate
89.8%
Best Reduction
44W / 3L / 0T
Record
34.5%
Avg Reduction
+3.5
Quality
93.6%
Win Rate
82.1%
Best Reduction
44W / 3L / 0T
Record
Gray = without Superstack. Colored = with Superstack. Lower is better.
Gray bar = without Superstack. Colored bars = with Superstack, per model.
Quality holds or improves across all difficulty levels — even on harder T3/T4 prompts.
Percentage of prompts where Superstack produced fewer tokens than bare.
Gray = without Superstack. Colored = with Superstack. Higher is better.
Per-prompt quality scores averaged across all models. Higher is better.
Output tokens per prompt averaged across all models. Hover for per-model breakdown.
Haiku
Sonnet
Opus
Opus 4.7
Each dot is one prompt. Dots below the red line = Superstack saved tokens.
188 data points. Click column headers to sort.
All tests used real Claude API calls with temperature=0 for reproducibility. Single-turn: each prompt tested A/B (bare vs Superstack). Multi-turn: 10 real coding tasks run with and without optimization. MCP: A/B comparison of flat tool definitions vs aggregator pattern. Token counts from API response metadata.
April 2026 · 230 total measurements
Full methodology on Results page