Split the confidence floor warning into two named findings - #242
Conversation
aveproject#238) Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com> Co-authored-by: Nicolai <245527909+predictor2718@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: chaksaray <15962335+chaksaray@users.noreply.github.com> Co-authored-by: Sankalp Gilda <sankalp.gilda@gmail.com> Co-authored-by: Empire Labs Pty Ltd <narko4u@gmail.com> Co-authored-by: narko4u <narko4u@users.noreply.github.com>
The composition of the declared vantage with the ceiling its engine set imposes was inline in derive(), reachable only as the front half of a composed verification_basis string. Extract it as derived_vantage() so a consumer asking only where an observation was made from has one supported way to ask, rather than splitting the composed value on an underscore or re-testing SUBSTRATE_ENGINES itself and becoming a second definition of the same predicate. No behaviour change: derive() composes the same two axes in the same order. The new test pins the name at its own level, since every end-to-end assertion through derive() would still pass if the read were withdrawn.
The check floored on engine-set cardinality, chosen when no measure of vantage existed. Running that predicate and the verification_basis derivation against each other over the same evidence_basis_engines field shows neither subsumes the other. With evidence_vantage=substrate, evidence_method=intercepted and confidence_baseline=0.9: engines cardinality derived basis two, neither substrate-capable silent artifact_intercepted one, substrate-capable fires substrate_intercepted one, not substrate-capable fires artifact_intercepted two, one substrate-capable silent substrate_intercepted Row one is a floor-level basis carrying a 0.9 that the check let through. Row two is the strongest basis available, flagged as a floor. Cardinality measures corroboration, how many independent sources agreed; the derivation measures vantage, where the observation was made from. Report both, separately. A vantage_floor finding is remedied by observing from somewhere the artifact cannot forge or suppress and declaring it; an independence_floor finding is remedied by adding a second source. A record may carry both, one, or neither, and the output names which, because a consumer told only that a record is weak cannot act on it. The vantage arm imports the derivation rather than recomputing it, and derives even where a record carries a stamped verification_basis: a second predicate over one field is the defect being repaired, and the stamp is the author's own copy, which a consumer-side check must not rest on. confidence_signal() returning a string is replaced by confidence_signals() returning named findings; every caller is updated and no compatibility path is kept. The external-authority note gains a second form for a record that has adopted the member and is now held at the floor by its own silence rather than by the vocabulary. A malformed engine member is no longer counted as an independent source, matching the reading the derivation already gives a malformed engine field. On the current corpus this turns 0 findings into 8 vantage_floor findings across 8 records, all of which declare a high confidence_baseline and no evidence_vantage at all. Still a soft warning; the exit code is untouched.
The consumer guide defined the floor as one predicate over evidence_basis_engines and evidence_kind_default, which is the definition the check no longer uses. Replace it with the two findings, the measured disagreement that separated them, and the remedy each one takes. Also corrects the AVE-2026-00074 section, which still described the record as carrying no external_authority member after it gained one, and restates the escalation condition against the enum-gap note rather than against the finding: a record that adopts the member can still owe an evidence_vantage declaration, and that is a producer's remaining step rather than evidence of a vocabulary gap.
|
The Clean break on the return shape is correct. A string can't carry two faults a consumer routes on differently, and a shim would have preserved an API that no longer describes the check. The Split, not the OR. No cutting back. One thing worth deciding, not blocking: nine lines where there was one is the fix working, but the vantage arm fires on all 8 until each author makes a judgment with no deadline attached. Worth knowing whether that count is expected to come down as records get stamped or sit at 8 and become background noise, soft warnings at volume stop being read. Not a reason to change this PR, just worth answering before it settles into whatever it settles into. Merging. |
Follows the question left open at the end of #214: whether check_confidence_signal.py should read verification_basis where present, or whether there is a real reason to keep the two independent.
This pull request answers neither. They are not one question at two levels of rigor. is_floor_basis tests cardinality, len(set(engines)) <= 1; engine_vantage tests capability, whether any member reaches substrate. Neither predicate contains the other, and all four combinations on origin/main show where they part:
Row 1 is a miss and row 2 fires on a record whose vantage is genuinely strong. The miss is live: of the 80 published records, 8 sit in the high band and every one derives artifact_reconstructed, because none declares evidence_vantage or evidence_method. None of the 8 is single-engine, so cardinality stays silent and the check currently reports that all 80 are consistent with their basis.
Reading verification_basis in place of cardinality would close the miss. It would also lose the corroboration signal, because a single substrate-capable engine has a strong vantage and one source. So this splits the check into two named findings with different remedies. A vantage floor is remedied by observing from somewhere closer. An independence floor is remedied by adding a second source. A record can carry either, both or neither, and the output says which.
What changed
is_floor_basis is gone.
vantage_floor_signalandindependence_floor_signalreplace it, assembled by confidence_signals with the high-band gate stated once. The return shape changes with them: confidence_signal returning a string or None becomes confidence_signals returning a list, because a single string cannot carry two faults a consumer routes on differently. There is no alias and no shim, and every caller is updated. The JSON output gains arecordskey, and count keeps meaning findings.derived_vantage is extracted in write_verification_basis.py so the check imports a name for the value it needs. The obvious move is to read a stamped verification_basis. The vantage arm derives instead: where a stamp exists validate_records.py already hard-fails a declaration the derivation refutes, so in a valid corpus the two agree, and where they could differ the stamp is what the author typed, which is issue #98 wearing a better field name.
semantic_inference folds into the vantage arm where a third finding would have split it, because the arms divide by remedy and an inference over meaning is a reading of artifact-produced content. The authority-probe note gained a second form, and says nothing about cardinality, since 00074 adopted external_authority in #218 and its ceiling now reaches substrate, so only the missing declaration holds it down. The old wording would have told that author to add a member the record already carries.
Consequences worth stating
The CI log for this check goes from one quiet line to nine. That is the fix working rather than a regression, and the exit code is unchanged; the check stays a soft warning.
Since silence derives the floor, the vantage arm fires on records that have not declared. That is correct for a consumer, because no substrate binding exists either way, and it does not penalise stating the floor: the high band is still required, so an honest artifact record with a mid baseline stays silent.
No record was edited to add evidence_vantage. The 8 findings are real, and each one is a producer-side judgement for that record's author to make.
Checks
CI's steps in order: validate_records 80 valid, validate_crosswalks 9 of 9, check_fixtures 80, check_confidence_signal 8 findings across 8 of 80, check_vulnerability_taxonomy clean, write_verification_basis all 80 agree with their derivation, pytest 450 passed. Tests went 435 to 450. Each of the three commits was tested independently from a clean snapshot.
Fourteen mutations were run against the new behaviour, one per claim the tests make, and all fourteen were killed.
If you would rather keep one signal and take the OR, say so and I will cut it back to that.