Skip to content

feat(evidence): expose failure artifact records - #153

Open
BenSheridanEdwards wants to merge 4 commits into
feat/supported-runner-configfrom
feat/failure-evidence-callback
Open

BenSheridanEdwards wants to merge 4 commits into
feat/supported-runner-configfrom
feat/failure-evidence-callback

Conversation

@BenSheridanEdwards

@BenSheridanEdwards BenSheridanEdwards commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a reporter-neutral onFailureEvidence config callback with project, file, fullName, retry attempt, and exact successfully produced PNG/XML paths
  • include that complete identity in a bounded full SHA-256 suffix so files do not collide across specs, projects, or retries
  • preserve public captureState(prefix): Promise<string> runtime compatibility: a source-read failure warns, degrades to "", then writes the screenshot/XML pair best-effort as before
  • keep internal captureStatePaths strict: a source-read failure rejects before screenshot/XML writes, so passing tests emit no callback and failed or incomplete captures emit no fictitious complete record or replacement failure
  • preserve built-in capture-first ordering and consumer afterTest behavior

Fixes #152

Depends on #151. This PR now contains repaired parent head 5ab8af8c6d0d04fb907cc527d3903d4004fdde37 via merge commit e244b8b5f0338cc977989a762bee36793925fab6; merge #151 first.

Proof

  • npm run check — pass
  • npm test — 237 passed, 0 failed
  • npm run build — pass
  • Generated project/device proof, or N/A: fresh /tmp/nativeproof-failure-evidence-consumer installed the exact npm pack tarball, typechecked a public nativeproof.config.ts using onFailureEvidence, loaded and invoked the callback, and ran packaged nativeproof --help; output included packed failure-evidence config proof: PASS and packed nativeproof --help: PASS. No device was required because this changes config translation, artifact identity, and hook association rather than device interaction.
  • Fresh post-restack packed consumer compiled a real EventEmitter reporter class, retry fields, and typed afterTest test/result fields including error; packaged CLI help passed.
  • Fresh compatibility-fix packed consumer installed exact tarball SHA-256 1b60928c45068c9dd46f5db239aa5d4c236917ade2edc16361a7ba688a2f3f53, typechecked public captureState(prefix): Promise<string>, simulated rejected getPageSource() through the packed runtime, and proved: public capture resolved "" plus PNG/empty XML and one warning; strict failure capture emitted zero structured records, skipped screenshot, preserved the original failed result by identity, and invoked consumer afterTest; packaged nativeproof --help passed.

Measured baseline → fixed comparison

Source baseline: PR #151 exact head ecb7f3cd752f605aa629f8c32aedadb543ef3372
Original feature head: 4f9fac8cf47b973297aba8ec7cabff449393ab14
Restacked fixed head: e244b8b5f0338cc977989a762bee36793925fab6
Acceptance fixed head: f79f0c15588d3ce878fa874198691760b2906592
Compatibility fixed head: 73c691b909153de7c81c271edb14412de64e9294

Same proof #151 baseline Fixed
Same title across two files, iOS/Android projects, and attempts 0/1 all four produced failure-Login-rejects_bad_password-d3b3f935 (uniqueNames: 1) regression produces four distinct prefixes/paths
Exact capture-path association injected capture returned exact PNG/XML paths, but the consumer hook could not observe them (capturePathsVisibleToConsumer: false) callback record equals the paths actually written by the injected capture
New regression tests on old behavior 18 passed, 2 failed diagnostics plus repaired runner regressions pass in test/config.test.ts; 235/235 full suite
Passing/capture-failure behavior no structured callback existed passing tests emit no callback; capture failure emits no record, does not throw, and passes the original failed result to consumer afterTest
Exact reported 32-bit collision (case-64394 / case-84739) both produce failure-Login-rejects_bad_password-6333da18; exact regression fails full SHA-256 names differ (...14898824…0718b2 / ...47c770c8…f903da), remain deterministic, and are 99 chars
Public captureState with rejected getPageSource() exact head f79f0c1… rejects; new compatibility regression fails 23/24 exact head 73c691b… warns, resolves "", and writes PNG/empty XML; focused regression passes 25/25 (five consecutive runs)
getPageSource() rejection public captureState warned/degraded to empty XML; no structured callback existed public captureState preserves that established fallback; internal captureStatePaths rejects before any file is reported, so the structured callback is skipped while the original failed result and consumer afterTest are preserved

Impact controls:

  • pre-edit GitNexus impact was LOW for composeAfterTest, failureEvidenceName, captureState, and buildWdioConfig; direct dependants were the config tests and runner-config translation
  • final gitnexus detect-changes --scope all: 6 files, 15 symbols, 2 affected config execution flows, MEDIUM; only the expected config/evidence/type/docs/tests surfaces changed
  • restack gitnexus detect-changes --scope all: 9 files, 33 symbols, 7 affected processes, HIGH; this is the combined feat(config): expose supported runner options #151 runner-config plus feat(evidence): expose collision-safe failure artifact records #152 failure-evidence stack, so full gates and a fresh packed-consumer compile were required and passed
  • acceptance-fix pre-edit impact for failureEvidenceName, captureStatePaths, and legacy captureState was LOW; final compare detected 14 symbols, 0 processes, LOW. GitNexus-generated AGENTS.md/CLAUDE.md count churn was restored before commit, leaving only src/evidence.ts and test/config.test.ts changed.
  • compatibility correction pre-edit impact for captureState and captureStatePaths was LOW. Final correction detect-changes --scope unstaged found 3 files, 5 symbols, 0 processes, LOW; full PR compare to feat(config): expose supported runner options #151 found 7 files, 26 symbols, 2 config flows, MEDIUM. Generated index churn was restored before commit.

Reviewer view

  • Smallest source-verified view for a non-trivial PR, or Not applicable with the reason for a trivial PR:
failed WDIO test
  └─ identity(project, file, fullName, attempt)
       └─ collision-safe prefix
            └─ read source (reject pair if unavailable) → capture PNG → capture redacted XML
                 └─ onFailureEvidence(exact record)
                      └─ consumer afterTest(original test result)
  • Source files, symbols, states, or commands that verify the view:
    • src/evidence.ts: captureStatePaths, captureState, failureEvidenceName
    • src/config.ts: FailureEvidence, composeAfterTest, buildWdioConfig
    • test/config.test.ts: exact known collision, unique identity, source-read rejection, exact-file callback, pass/no-callback, and failed-capture ordering cases
    • npm run check && npm test && npm run build
    • fresh packed consumer at /tmp/nativeproof-pr153-compat-packed (tarball SHA-256 above) exercises both public fallback and strict callback suppression

Risk

  • onFailureEvidence is optional and reporter-neutral; no reporter package, Allure API, or NativeProof test DSL was added.
  • Callback and capture errors are best-effort and cannot block the existing consumer afterTest; successful capture still occurs before both callbacks.
  • Public captureState(prefix): Promise<string> keeps its established best-effort runtime behavior: source-read failure warns, degrades to "", and continues to screenshot/XML writes. Internal captureStatePaths remains strict; runner composition swallows its source failure, emits no structured record, and still invokes the consumer hook with the original result.
  • Conflict resolution preserved the diagnostic callback behavior while adopting feat(config): expose supported runner options #151's supported WDIO reporter-class and exact Frameworks.Test/Frameworks.TestResult hook types.

Notes

  • Full device coverage not run: no device interaction changed, and no device or screenshot proof was fabricated. Unit, full-suite, build, and fresh packed-consumer paths were exercised.
  • No merge, npm release, or deployment performed.

PR Proof Law

  • I followed .agents/DEFINITION_OF_DONE.md and .agents/skills/pr-inline-screenshot-proof/SKILL.md.
  • Screenshots are committed and embedded inline with ![alt](...png?raw=1), or the proof section says Not applicable with the technical reason.
  • The PR body has no bare screenshot links, local paths, relative paths, or proof placeholders.

Screenshot proof: Not applicable — this is a config/type/artifact-path association change with no rendered UI. The source-verified call-order view and executable baseline/fixed proofs above are the appropriate evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant