Derive verification_basis from a record's own evidence axes (issue #98) - #214
Conversation
Add external_authority to evidence_basis_engines, for the case where a party outside the observed artifact was queried and returned a determinate answer: a package registry, RDAP, a forge's user API. The six existing members all run over content the artifact produced, so a record whose finding rests on an outside answer had no value to write and its author wrote the nearest one. Add two producer-written axes beside it. evidence_vantage says whether every input the evidence depends on was obtained where the artifact could neither forge nor suppress it. evidence_method says whether the evidence was captured as the events occurred or rebuilt from state afterwards. Both are closed two-value vocabularies, both are taken from the weakest input, and the weaker value of each is a claim a producer may always truthfully make, so stating it honestly costs nothing. Add verification_basis as the composition of those two with the vantage the engine set implies. It is derived rather than authored; a record that declares one has the declaration checked against the derivation, which is what keeps it from becoming a second self-reported number. Refs aveproject#98 Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
A check that reads confidence_baseline is reading a self-report, only a better organised one. What changes that is the producer stating where an observation was made from, so verification_basis is computed from the record's own closed axes and never authored. The engine set is a ceiling and the declared vantage is the claim, and the derived value is the weaker of the two. So a record cannot raise its own basis by asserting a vantage its evidence has no way to occupy, and cannot be credited with one it never claimed. Absence reads as the floor on both axes, and a value outside either closed vocabulary reads as the floor rather than as a new stronger rung. --write refuses any record whose file is not already in the script's own serialisation, because 73 of the 80 records differ from it in escaping or whitespace alone and stamping one word into them would produce a diff nobody can review, with the real change hidden inside it. Canonicalising the corpus stays separate work. Refs aveproject#98 Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
A derived field a producer may also declare is only worth carrying if the declaration can be falsified, so validate_records now recomputes it and fails on a mismatch. Understating is a mismatch too: the field states what the derivation computes, not what its author would prefer it to say. This is a hard error and not a soft warning because nothing about it needs a human glance. A record whose declared basis disagrees with its own axes is refuted by its own contents, unlike the researcher warning beside it, where some records are genuinely original cataloguing. It goes inside validate_records rather than beside it because that is the script CI already runs, and a check nothing invokes reports nothing. Tests cover both: one drives the validator over a contradicted record and turns red when the call is removed, and its negative control drives the same record without the declaration so a validator failing everything cannot satisfy the first. Refs aveproject#98 Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
A producer guide for the two axes, the derived field, and the new engine member, written around the trap the design turns on: the weaker value of each axis has to read as a floor a producer may always truthfully state. If it reads as a confession, honest authors avoid it, values drift upward, and the axis means nothing within a year. Also names the derivation script in both lists of local checks, next to the ones already there, so a contributor whose record states a vantage finds out locally rather than in review. Refs aveproject#98 Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
|
Traced derive() by hand across the three cases that matter: declared-weak-with-strong-engines (correctly stays weak), declared-strong-with-weak-engines (correctly forced to floor, the load-bearing case), and declared-strong-with-strong-engines (correctly holds). The ceiling-caps-the-claim model is real, not just described that way. Worth naming directly: two places in this test suite exist because of mistakes found in someone else's PR days ago, now applied to your own before anyone had to find them. Pinning declared_vantage/declared_method directly rather than only through derive() is the identical fix to the unused set() call from #213. The wiring-gap negative control on validate_records.py's own integration is the identical fix to the check that ran once inside a passing pytest call and printed nowhere. That's real, demonstrated internalization, not a coincidence of good practice. Hard-fail vs. #213's soft-warn is correctly reasoned, not just inconsistent styling: a confidence/basis disagreement might still be an honest finding poorly represented by the current vocabulary, which is exactly what 00074 turned out to be. A declared verification_basis contradicting its own record's axes is arithmetic the record refutes, not a judgment call. Different epistemic status, correctly different enforcement. One real question worth deciding now rather than later: this PR builds exactly the vantage-based mechanism #213's review said the confidence-signal floor should have been. That leaves two checks answering related questions with different rigor, confidence_signal.py still floors on cardinality, verification_basis now floors on actual vantage. Once this merges, should check_confidence_signal.py read verification_basis directly where present, falling back to cardinality only where absent? Or is there a real reason to keep them independent? Worth a stated answer, not two mechanisms coexisting by default. Merging. That last question is worth its own follow-up, not a blocker here. |
|
This merged before your detailed review landed, if you still want to go through it the way you offered, that's genuinely welcome even post-merge, the same standard applied when #213 merged ahead of a second confirmation pass. Nothing here is closed off just because it's already in. |
|
Thank you for tracing derive() by hand across the three cases, and for naming the two places the tests came from. Sorry this took a few days. You asked for a stated answer. Here it is, and it starts with what I had wrong. I assumed the two checks asked one question at two levels of rigor, so verification_basis would simply subsume check_confidence_signal.py. They do not. is_floor_basis tests cardinality, len(set(engines)) <= 1. engine_vantage tests capability, whether any member reaches substrate. Neither predicate contains the other. All four combinations on origin/main, with declared vantage substrate, method intercepted, confidence 0.9:
Rows one and two are where they part. Row one is a miss: the basis floors and the check says nothing, because two names satisfy cardinality whatever those names can do. Row two fires on a record whose vantage is genuinely strong, because a set of one counted as weak. The miss is not hypothetical, though the published records reach it by a different route than row one. Of the 80 published records, 8 sit in the high band and every one derives artifact_reconstructed, because none of them declares evidence_vantage or evidence_method at all. None of the 8 is single-engine either, so cardinality stays silent and the check prints that all 80 are consistent with their basis. The reason is that check_confidence_signal.py predates verification_basis. Cardinality was standing in for weak evidence because nothing better existed yet. Now that a real vantage measure does exist, what cardinality was actually measuring shows up clearly, and it is corroboration. So reading verification_basis in place of cardinality would fix the miss and lose something real, because a single substrate-capable engine has a strong vantage and still rests on one source. OR-ing the two would close the miss and leave two different findings collapsed under one signal, so a consumer still could not tell which weakness they had. What I think is right is to split them, because each one has a different fix. A vantage floor says a record was observed from too far away and you fix it by observing from somewhere closer, while an independence floor says one source carried the whole claim and a second source closes it. A record can carry either, both or neither. The signal should say which. That also settles row two. A single substrate-capable engine at high confidence is a finding, and the weakness is independence while the check reports vantage. The current behaviour spots the right record and names the wrong fault. I have this built against origin/main with tests, and I will open it as a pull request so you can read the diff itself. If you would rather keep one signal and take the OR, say so and I will cut it back. The detailed review I owe you comes next, and it goes against whichever shape you land on. |
|
rows 1 and 2 are exactly where the predicates come apart, and neither contains the other. This corrects what I agreed to on #219, and specifically how: I flagged row 1 there myself and read it as "cardinality is strictly weaker, therefore replaceable." Wrong conclusion. It isn't weaker, it's measuring independence rather than vantage, and row 2 is the case I missed that proves it. The subsumption claim was mine as much as @narko4u's. The 8-record finding is the part that settles it, the check prints all-clear today while 8 high-band records derive Split, not OR. Two weaknesses with different fixes shouldn't share one signal, and "spots the right record and names the wrong fault" is the right diagnosis of row 2. One question worth settling before the diff, because it decides whether the vantage finding reports 8 or 0. All 8 of those records derive artifact because they declare no Options as I see them: fire on both with distinct messages, fire only on a stamped artifact declaration and stay silent on unstamped records until they are stamped, or fire on both identically and accept that the count tracks stamping progress for now. I lean toward the first, an unstamped record is a real gap worth surfacing, just not the same gap as a record that looked and declared honestly. Your call, you have the producer side in view better than I do. No worries on the delay. |
This adds the producer half of #98. Two closed axes go into the record schema.
evidence_vantagetakes substrate or artifact, andevidence_methodtakes intercepted or reconstructed. Both are optional and producer-written, and both are taken from the weakest input a claim rests on. Beside them sitsverification_basis, which composes the two, andscripts/write_verification_basis.pycomputes it rather than an author typing it.validate_records.pyrecomputes any declared value and fails on a mismatch, the pin_status shape from #171.That last part is the only reason a declaration is worth carrying.
confidence_baselineis a float its author assigns, and a check that reads it is still reading a self-report. What changes that is a producer stating structurally what it observed, in a vocabulary a validator can recompute from the record's own contents.The mechanic is the part worth reviewing. The engine set is a ceiling, the declaration is the claim, and the derived value is the weaker of the two. The pattern, yara, semgrep, llm and magika engines all read content the artifact produced. A record detected only by those cannot reach substrate whatever it declares. The sandbox engine observes execution from outside the artifact, and the new
external_authoritymember asks a party outside it entirely. Either of those can reach substrate, so a producer cannot raise its own basis by asserting a vantage its evidence cannot occupy.That member is load-bearing rather than decorative. Before it existed, a record whose finding came from asking an outside party had its ceiling pinned at artifact. The enum had no member for that rung. No honest author could state the true value, however the evidence was actually obtained. Adding the member raises no record on its own, since an author still declares the vantage. It makes the true statement available.
The floor is what keeps the axes honest. Artifact and reconstructed are the weaker value of each axis, and a producer may always truthfully state either. Absence reads as the floor on both, so silence is never credited as the stronger claim. A value outside the closed vocabulary reads as the floor too, so a typo cannot raise a record. A non-list engine field reaches nothing, for the same reason. Understating fails the check as well, so the field cannot quietly opt out of a stronger basis.
That choice is deliberate. If the weaker value read as a confession, honest authors would avoid it. The values would drift upward, and the axis would die inside a year.
This deliberately does not touch
AVE-2026-00074. Adding the external_authority member to that record is what clears the fixture in #213. So it belongs in its own commit after that PR merges, not as a side effect of the schema landing. I trial-merged the two branches locally and they do not fight. There is no conflict and the combined suite is green. The consumer check still fires exactly once on that record, and the derivation passes on all 80 records.One scoping note on the writer. In write mode it refuses any record that is not already byte-identical to its own serialisation. 73 of the 80 records on main differ from it in escaping or whitespace alone. Stamping a one-word field into them would produce a diff nobody can review, and would hide the real change inside it. Those records are refused rather than rewritten. Corpus canonicalisation is already named as separate work in #98, and it stays that way.
All four repository checks pass on this branch: validate_records.py, validate_crosswalks.py, check_fixtures.py, and the pytest suite at 417 passing.
The caveat from the issue thread still holds, and I would keep it in the guide verbatim. The derivation names a vantage, and it says where an observation was made from and how. It does not certify that a record is right, and it does not create an audit trail on its own.