Decision: leaving 3.4% of the reported saving with unverified provenance

[HackPoint]

What

full_tokens is the denominator of every efficiency claim Lumen makes, and token_source records whether that number came from the tokenizer or from a bytes/4 estimate. 96.6% of the reported saving rests on a real tokenizer count. The remaining 3.4% is rows written before the token_source column existed, so they cannot be classified retrospectively. This records the decision to leave those rows unclassified rather than backfill them, and closes as wontfix so the reasoning stays searchable.

Evidence

• Provenance share over the live ledger: 96.6% of the reported saving comes from a measured count, 3.4% does not. Printed and asserted by crates/lumen-mcp/tests/efficiency.rs, test the_recorded_savings_are_priced_and_their_provenance_is_known — it prints the share, then asserts it stays above 80.0, so a regression that stops recording token_source fails instead of quietly eroding the figure.
• The unknown is disclosed, not inferred: unverified_provenance_rows counts every row where token_source IS NULL OR token_source <> 'measured' — crates/lumen-stats/src/lib.rs:494.
• Per-provenance breakdown of the same ledger: the "Token provenance" section of .claude/lumen_report.sh.

The ledger is append-only by design. Rewriting the pre-column rows would replace a disclosed unknown with an undisclosed guess — the same mistake as the bytes/4 estimates that made a screenshot read 32,000 tokens.

The cost of this choice is real. The disclosure is a row count, not a share of the saving, so a reader sees how many rows are unclassified without seeing what fraction of the headline they carry. And the test floor is 80.0 while the measured share is 96.6, so the figure can drift a long way before anything fails.

What would change this

• The printed share falls below the 80.0 floor, or drops between releases. The response is to fix metering, not to soften the wording.
• The pre-column rows become classifiable from data already stored, so a recount is a measurement rather than a guess.
• A reported efficiency figure ships without unverified_provenance_rows alongside it, making the unknown invisible.
• The unverified fraction is needed in tokens rather than rows. Add a second aggregate; do not rewrite rows.

How to re-measure

# From the repository root. Requires a metering ledger on this machine;
# set LUMEN_DB=/path/to/lumen.db if the app has never run here.
cargo test -p lumen-mcp --test efficiency \
 the_recorded_savings_are_priced_and_their_provenance_is_known -- --nocapture

# Row and token counts per provenance value, same ledger
./.claude/lumen_report.sh

Expect a line reading of that saving, NN.N% rests on a tokenizer count rather than bytes/4, and a pass while NN.N exceeds 80.0. With no readable ledger the test returns early and passes without printing — that is a skip, not a confirmation. The report script prints one row per token_source value; unknown is the pre-column rows.