Proof categories
Each evidence lane has a separate boundary.
Raw provider meter differences, provider-native caching, CostLayers provider-call avoidance, and local context estimates are not interchangeable claims.
Legacy 10-row API artifact
Records a mixed-arm raw meter delta and 10 historical quality labels. It is quarantined, provider-native effects were not isolated, and no part of the aggregate is counted as CostLayers-attributed savings.
Repeated-cache provider-call avoidance
Separately proves repeated CostLayers gateway calls can avoid upstream provider calls. The old partial artifact has provider_avoided=0.
Local context matrix
Proves public command/context behavior and quality gates. Does not prove provider-dollar savings or autonomous code-edit success.
Still missing
Real customer invoice export, full live task matrix, and third-party or raw-enough review remain outside the current public proof set.
Frozen inputs
Ten external repos, three task types, four baselines.
The public context matrix excludes VS Code and oversized repos for this proof. It is complete as local command/context evidence; the full live autonomous Codex API matrix remains pending.
External repos
Task types
Baselines
Public evidence rows
Every savings number keeps its boundary.
Raw meter totals, provider-native effects, and verified CostLayers-attributed savings are reported separately. Local context artifacts are not promoted into provider-dollar claims.
| Evidence | Repos and tasks | Baselines | CostLayers result | Quality status | Savings classification | Provider avoided | Bound hash | Claim boundary |
|---|---|---|---|---|---|---|---|---|
| 10-repo public context benchmark complete published matrix |
requests, flask, click, express, gin, itsdangerous, markupsafe, packaging, jinja, httpx; three task classes | plain, codex-codegraph, codex-codebase-memory, costlayers | 120/120 rows OK; 30/30 CostLayers rows OK | 120/120 quality gates OK | Not claimed | Not claimed | 8427f5e93ea072f4e0488cfc16d7a4fb4be8621e065df9e0feffe9df5f0361c3 | Public local command/context benchmark only. Not verified API invoice proof and not autonomous code-edit success proof. |
| Legacy 10-row live Codex artifact quarantined; non-promotable |
requests, flask, click partial; architecture-qa / repo-qa rows | plain Codex, Codex+CodeGraph, Codex+codebase-memory, CostLayers full mode | Historical raw run retained for audit; stopped at 10 rows by operator | 10 recorded labels; not current verification | Legacy mixed-arm meter field excluded from verified savings; provider-native portion not isolated; $0 counted as verified CostLayers-attributed savings | 0 in these one-off rows | d77731d4adcd2a3afa2e9ddd1481a447343a1a2a4baa51e6c7df14a66c885dea | Immutable legacy file hash only. Partial, unbalanced, no CostLayers attribution receipts, provider_avoided=0, and best-in-world=false. Promotion is prohibited. |
| Strict external context matrix complete context matrix |
requests, flask, click, express, gin; all five frozen task types | plain, codex-codegraph, codex-codebase-memory, costlayers context commands | 100/100 rows OK; 25/25 CostLayers rows OK | 100/100 quality gates OK | Not claimed | Not claimed | 0fccccde649c3906bd6c4258698edb7116afa395809a68ac90753ddcfd437263 | Command/context benchmark only. Not verified API invoice proof and not autonomous coding-agent task-success proof. |
| Provider-call avoidance scale smoke complete smoke |
Direct repeated gateway API requests | first routed request vs repeated routed requests | 12 requests, 1 provider call, 11 provider calls avoided | 11 quality-safe avoided rows | $0 verified CostLayers-attributed; $0.001034 aggregate meter excluded because request attribution receipts are absent | 11 | dfd6c486e46f6a4c5542e2c871583bc894373f205d2772f36184a30f00c93435 | Proves repeated gateway response-cache avoidance only. Not the full autonomous Codex benchmark. |
| Legacy API meter micro-proof attribution excluded |
Gateway micro-proof, repeated API request | first routed request vs repeated routed request | 2 routed API rows, 1 provider call avoided | legacy label; not current verification | $0 verified CostLayers-attributed; $0.000082 aggregate meter excluded and verified provider avoidance is 0 | 1 | 28f1ed2913bc188d2f57fb55d17689145d33283328ac7b66c0e4d779028b25f2 | Quarantined legacy meter file. It lacks current request-level attribution receipts, is not customer invoice proof, and cannot be promoted. |
| Memory backend integration proof complete backend proof |
django, tokio, excalidraw backend routing receipts | CodeGraph and codebase-memory through CostLayers | 3/3 repo receipts with both local backends available | not task-quality proof | Not claimed | Not claimed | 2b8e5cd8dd3812f291282a9128e75aa104531b9ed2be57770884d5fd4473903c | Local context-routing/backend integration proof only; no provider-dollar or task-quality claim. |
| Full live matrix attempt blocked |
100 planned rows; first requests architecture-qa baseline set attempted | plain, codex-codegraph, codex-codebase-memory, costlayers | 4 attempted rows failed before provider invoice counters due provider quota exhaustion | no accepted live rows from this attempt | $0 counted | 0 counted | 47993d43b11125bab93caa8c59264e2ff4bf538d4737ecf62f6051d7c4883aff | Quarantined failed-attempt file hash only. It contains no accepted live rows and cannot be promoted. |
Artifact links
Current packets and legacy quarantine stay separate.
Current evidence links keep their narrow boundaries. Legacy raw runs, failed runs, JSONL, and screenshots are accessible only through the non-promotable hash registry.
Raw runs and screenshots
Hash-bound historical and failed artifacts; current verified proof is false and promotion is prohibited.
registry policy: prohibited
Open descriptor10-repo context benchmark
Ten repos, three task types, four baseline labels, 120 rows.
8427f5e93ea072f4e0488cfc16d7a4fb4be8621e065df9e0feffe9df5f0361c3
Open JSONSmall strict matrix
Five repos, five task types, four baseline labels, 100 rows.
0fccccde649c3906bd6c4258698edb7116afa395809a68ac90753ddcfd437263
Open JSONRepeated gateway smoke
12 repeated requests, 11 provider calls avoided.
dfd6c486e46f6a4c5542e2c871583bc894373f205d2772f36184a30f00c93435
Open JSONLocal backend routing
CodeGraph and codebase-memory routing through CostLayers on three repos.
2b8e5cd8dd3812f291282a9128e75aa104531b9ed2be57770884d5fd4473903c
Open JSONReproducible demo
Verify public labels and hashes locally.
No separate public sample repo is attached. Use the local recipe and quarantine registry to verify labels and immutable hashes without promoting raw legacy data.
node benchmarks/world-best/claim-consistency-smoke.js
node benchmarks/world-best/quality-safe-proof-fixture-smoke.js
node -e 'const r=require("./proof_artifacts/legacy-artifact-registry.json"); console.log(r.registry_status, r.policy.promotion_status, r.artifacts.length)'Expected labels include claim consistency smoke passed, quality-safe proof fixture smoke passed, active_quarantine, and prohibited.
The consistency smoke recomputes registered file-byte hashes. Internal legacy certificate fields are not file hashes and do not authorize promotion.
Claim boundaries
Current claims are deliberately narrow.
The allowed public wording is that CostLayers has a 10-repo public context benchmark, a legacy partial 10-row API artifact with a raw mixed-arm meter delta, and separate repeated-cache provider-call avoidance proof.
No best-in-world claim
The frozen checklist still requires the full live autonomous Codex API matrix, full-matrix savings evidence, fresh-install verification, customer invoice export, and independent review.
No unattributed savings counted
Provider-native, estimated, raw, and unverified totals stay separate from verified CostLayers-attributed savings.
No ChatGPT subscription claim
The evidence applies to API-billed Codex/OpenAI traffic routed through CostLayers, not flat subscription billing.
old_partial_artifact: legacy_verified_field_accepted=false; verified_costlayers_attributed_saved_cost_usd=$0.000000; provider_avoided=0; best-in-world=false
The old 10-row paid live Codex artifact is API-mode meter evidence, but it is partial and unbalanced, aggregates all benchmark arms, and does not separate provider-native cached-input discounts from CostLayers actions.
The public 10-repo context matrix is current evidence for command/context behavior only; it is not verified API invoice proof and not autonomous code-edit success proof.
Missing proof checklist
External proof gaps are still visible.
These gaps are not papered over by the public artifacts.
Real customer invoice export: missing. This requires customer data and cannot be created without customers.
Full live task matrix: missing. The current paid live API proof is 10 rows, not the full matrix.
Third-party or raw-enough review: missing. Public artifacts are redacted and hash-only; independent review still needs controlled access to enough raw evidence.