Skip to content

Split the confidence floor warning into two named findings - #242

Merged
chaksaray merged 4 commits into
aveproject:developfrom
astrogilda:fix/split-confidence-floor-findings
Sep 2, 2026
Merged

Split the confidence floor warning into two named findings#242
chaksaray merged 4 commits into
aveproject:developfrom
astrogilda:fix/split-confidence-floor-findings

Conversation

@astrogilda

Copy link
Copy Markdown
Contributor

Follows the question left open at the end of #214: whether check_confidence_signal.py should read verification_basis where present, or whether there is a real reason to keep the two independent.

This pull request answers neither. They are not one question at two levels of rigor. is_floor_basis tests cardinality, len(set(engines)) <= 1; engine_vantage tests capability, whether any member reaches substrate. Neither predicate contains the other, and all four combinations on origin/main show where they part:

# engines confidence check derived basis
1 two, both short of substrate silent artifact_intercepted
2 one, substrate-capable fires substrate_intercepted
3 one, without substrate reach fires artifact_intercepted
4 two, one substrate-capable silent substrate_intercepted

Row 1 is a miss and row 2 fires on a record whose vantage is genuinely strong. The miss is live: of the 80 published records, 8 sit in the high band and every one derives artifact_reconstructed, because none declares evidence_vantage or evidence_method. None of the 8 is single-engine, so cardinality stays silent and the check currently reports that all 80 are consistent with their basis.

Reading verification_basis in place of cardinality would close the miss. It would also lose the corroboration signal, because a single substrate-capable engine has a strong vantage and one source. So this splits the check into two named findings with different remedies. A vantage floor is remedied by observing from somewhere closer. An independence floor is remedied by adding a second source. A record can carry either, both or neither, and the output says which.

What changed

is_floor_basis is gone. vantage_floor_signal and independence_floor_signal replace it, assembled by confidence_signals with the high-band gate stated once. The return shape changes with them: confidence_signal returning a string or None becomes confidence_signals returning a list, because a single string cannot carry two faults a consumer routes on differently. There is no alias and no shim, and every caller is updated. The JSON output gains a records key, and count keeps meaning findings.

derived_vantage is extracted in write_verification_basis.py so the check imports a name for the value it needs. The obvious move is to read a stamped verification_basis. The vantage arm derives instead: where a stamp exists validate_records.py already hard-fails a declaration the derivation refutes, so in a valid corpus the two agree, and where they could differ the stamp is what the author typed, which is issue #98 wearing a better field name.

semantic_inference folds into the vantage arm where a third finding would have split it, because the arms divide by remedy and an inference over meaning is a reading of artifact-produced content. The authority-probe note gained a second form, and says nothing about cardinality, since 00074 adopted external_authority in #218 and its ceiling now reaches substrate, so only the missing declaration holds it down. The old wording would have told that author to add a member the record already carries.

Consequences worth stating

The CI log for this check goes from one quiet line to nine. That is the fix working rather than a regression, and the exit code is unchanged; the check stays a soft warning.

Since silence derives the floor, the vantage arm fires on records that have not declared. That is correct for a consumer, because no substrate binding exists either way, and it does not penalise stating the floor: the high band is still required, so an honest artifact record with a mid baseline stays silent.

No record was edited to add evidence_vantage. The 8 findings are real, and each one is a producer-side judgement for that record's author to make.

Checks

CI's steps in order: validate_records 80 valid, validate_crosswalks 9 of 9, check_fixtures 80, check_confidence_signal 8 findings across 8 of 80, check_vulnerability_taxonomy clean, write_verification_basis all 80 agree with their derivation, pytest 450 passed. Tests went 435 to 450. Each of the three commits was tested independently from a clean snapshot.

Fourteen mutations were run against the new behaviour, one per claim the tests make, and all fourteen were killed.

If you would rather keep one signal and take the OR, say so and I will cut it back to that.

chaksaray and others added 4 commits August 30, 2026 08:16
aveproject#238)

Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
Co-authored-by: Nicolai <245527909+predictor2718@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: chaksaray <15962335+chaksaray@users.noreply.github.com>
Co-authored-by: Sankalp Gilda <sankalp.gilda@gmail.com>
Co-authored-by: Empire Labs Pty Ltd <narko4u@gmail.com>
Co-authored-by: narko4u <narko4u@users.noreply.github.com>
The composition of the declared vantage with the ceiling its engine set
imposes was inline in derive(), reachable only as the front half of a
composed verification_basis string. Extract it as derived_vantage() so a
consumer asking only where an observation was made from has one supported
way to ask, rather than splitting the composed value on an underscore or
re-testing SUBSTRATE_ENGINES itself and becoming a second definition of
the same predicate.

No behaviour change: derive() composes the same two axes in the same
order. The new test pins the name at its own level, since every
end-to-end assertion through derive() would still pass if the read were
withdrawn.
The check floored on engine-set cardinality, chosen when no measure of
vantage existed. Running that predicate and the verification_basis
derivation against each other over the same evidence_basis_engines field
shows neither subsumes the other. With evidence_vantage=substrate,
evidence_method=intercepted and confidence_baseline=0.9:

  engines                          cardinality   derived basis
  two, neither substrate-capable   silent        artifact_intercepted
  one, substrate-capable           fires         substrate_intercepted
  one, not substrate-capable       fires         artifact_intercepted
  two, one substrate-capable       silent        substrate_intercepted

Row one is a floor-level basis carrying a 0.9 that the check let through.
Row two is the strongest basis available, flagged as a floor. Cardinality
measures corroboration, how many independent sources agreed; the
derivation measures vantage, where the observation was made from.

Report both, separately. A vantage_floor finding is remedied by observing
from somewhere the artifact cannot forge or suppress and declaring it; an
independence_floor finding is remedied by adding a second source. A record
may carry both, one, or neither, and the output names which, because a
consumer told only that a record is weak cannot act on it.

The vantage arm imports the derivation rather than recomputing it, and
derives even where a record carries a stamped verification_basis: a second
predicate over one field is the defect being repaired, and the stamp is
the author's own copy, which a consumer-side check must not rest on.

confidence_signal() returning a string is replaced by confidence_signals()
returning named findings; every caller is updated and no compatibility
path is kept. The external-authority note gains a second form for a record
that has adopted the member and is now held at the floor by its own
silence rather than by the vocabulary. A malformed engine member is no
longer counted as an independent source, matching the reading the
derivation already gives a malformed engine field.

On the current corpus this turns 0 findings into 8 vantage_floor findings
across 8 records, all of which declare a high confidence_baseline and no
evidence_vantage at all. Still a soft warning; the exit code is untouched.
The consumer guide defined the floor as one predicate over
evidence_basis_engines and evidence_kind_default, which is the definition
the check no longer uses. Replace it with the two findings, the measured
disagreement that separated them, and the remedy each one takes.

Also corrects the AVE-2026-00074 section, which still described the
record as carrying no external_authority member after it gained one, and
restates the escalation condition against the enum-gap note rather than
against the finding: a record that adopts the member can still owe an
evidence_vantage declaration, and that is a producer's remaining step
rather than evidence of a vocabulary gap.
@chaksaray

Copy link
Copy Markdown
Contributor

The derived_vantage choice is the part I'd have gotten wrong, I'd have read the stamped verification_basis, and your reasoning against it is right: validate_records already hard-fails a refuted declaration, so in a valid corpus they agree, and in the case where they could differ the stamp is what an author typed. Reading it would put a self-report back inside the check built to catch self-reports.

Clean break on the return shape is correct. A string can't carry two faults a consumer routes on differently, and a shim would have preserved an API that no longer describes the check.

The semantic_inference fold is the right call for the reason you give, the arms divide by remedy, and an inference over meaning is still a reading of artifact-produced content. Consistent with the split's own principle rather than an exception to it.

Split, not the OR. No cutting back.

One thing worth deciding, not blocking: nine lines where there was one is the fix working, but the vantage arm fires on all 8 until each author makes a judgment with no deadline attached. Worth knowing whether that count is expected to come down as records get stamped or sit at 8 and become background noise, soft warnings at volume stop being read. Not a reason to change this PR, just worth answering before it settles into whatever it settles into.

Merging.

@chaksaray
chaksaray changed the base branch from main to develop September 2, 2026 16:01
@chaksaray
chaksaray merged commit d997b25 into aveproject:develop Sep 2, 2026
6 checks passed
chaksaray added a commit that referenced this pull request Sep 2, 2026
Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants