Add a cross-repo fleet-conformance gate so emitter drift is caught

[stranske]

Why
The only fleet tests validate Workflows-local fixtures (tests/scripts/test_langsmith_fleet.py:9,13,35 all load from tests/fixtures/langsmith_fleet/{valid,invalid}.ndjson); nothing validates a live consumer artifact. As a direct result, Portable-Alpha has been emitting records that fail the canonical validator on every line (see Issue #3) with no signal. The contract is "contract-first" (docs/contracts/langsmith-observability-contract.md) but has zero runtime enforcement that consumer artifacts actually conform, so emitter drift is invisible until a human runs the validator by hand.

Scope

• A scheduled/dispatch conformance check in Workflows that, for each registered repo, validates its latest langsmith-fleet.ndjson against scripts/langsmith_fleet.py + registry and reports per-repo status (missing / invalid / stale / valid) plus the first error.
• May be folded into Issue Clean up validate_fast.sh Phase 4 TODOs and remove project-specific references #1's job or implemented as a sibling maint-8x-* workflow.
• Warning-only by default; hard-block only behind an explicit, documented approval input.

Non-Goals

• Not a redesign of the validator; reuse scripts/langsmith_fleet.py and summarize_fleet_records as-is.
• Not a gate on consumer-repo CI — this is a fleet-level check that lives in Workflows (consumer-repo maintenance belongs to the consumer per REPO_REVIEW_PROCESS.md:217).
• A workflow that always reports green, or that only re-checks the local fixtures, does not satisfy this issue. The gate must validate the actual per-repo artifacts (or honestly report missing).

Tasks

• Add a job that, for each repos[].repo in config/langsmith_fleet_registry.json, obtains the repo's latest langsmith-fleet.ndjson (reusing the transport built in Issue Clean up validate_fast.sh Phase 4 TODOs and remove project-specific references #1) and runs python scripts/langsmith_fleet.py <artifact> --registry config/langsmith_fleet_registry.json --summary --format json.
• Parse the summary into a per-repo status + first_error row (the summarize_fleet_records output already provides status and first_error, scripts/langsmith_fleet.py:366-440).
• Default the gate to warning-only (emit ::warning per non-valid repo); gate any hard-block behind a documented workflow_dispatch boolean input (e.g. enforce_block, defaulting false).
• Upload a machine-readable conformance report artifact (per-repo repo/surface/status/first_error).
• Document the gate and its warning-vs-block policy in docs/contracts/langsmith-observability-contract.md under the existing "Validation Expectations" (line 67) and "Dashboard Status Contract" (line 97) sections.

Acceptance Criteria

• Failing test (drift detection): add a tests/scripts/ case that feeds the gate-parsing helper a non-conformant record in PA's current shape (top-level schema/generated_at, missing surface/run_id/github_issue) and asserts the resulting status for that repo/surface is invalid with a non-empty first_error; a conformant record yields valid. This test fails if the gate ever silently passes malformed input.
• Documented live-verification gate: a dispatch run reports each registered repo's status and the report distinguishes missing (no artifact uploaded yet — the honest current state for most repos) from invalid (uploaded but malformed). Capture the run URL and the conformance-report artifact contents in the PR.

Evidence

• Workflows/tests/scripts/test_langsmith_fleet.py:9,13,35 — every assertion loads local fixtures; no live consumer artifact is validated.
• Canonical validator against PA's artifact fails every line: 'schema_version' is a required property, 'surface' is a required property, 'run_id' is a required property, 'github_issue' is a required property, stranske/Portable-Alpha-Extension-Model/ is not in registry (verified by running scripts/langsmith_fleet.py on Portable-Alpha-Extension-Model/artifacts/langsmith/langsmith-fleet.ndjson).
• No consumer workflow uploads the artifact (grep of all 7 consumer .github/workflows/ for langsmith-fleet returns none), so today's honest status for most repos is missing.

Implementation Notes

• summarize_fleet_records (scripts/langsmith_fleet.py:366-440) already classifies into missing/invalid/stale/valid and surfaces first_error per row — reuse it; do not re-implement classification.
• Correction to prior draft: there is no docs/keepalive/Observability_Contract.md and no "§6A" in this repo (verified absent). The warning-vs-block precedent to mirror is the existing ::warning-on-lookup-failure pattern already in maint-80-langsmith-metrics-dashboard.yml (near line 89). Use that style; do not cite a non-existent doc.

---

---

Filed from the 2026-05-29 dev-process review.

[stranske]

Closer merged PR #2192 and applied verify:compare. Reopening for verifier sequencing; this issue should remain open until a durable verifier/provider-comparison PASS is available and no disposition debt remains.

[stranske]

Closer verifier-disposition audit for merged PR #2192:

• Refuted the truncated-diff concerns against current main: tests/scripts/test_langsmith_fleet_conformance.py explicitly exercises the Portable-Alpha malformed shape (schema/generated_at, missing surface/run_id/github_issue) and asserts the PA row is invalid with first_error.
• Refuted the transport/report concerns against current main: .github/workflows/maint-81-langsmith-fleet-conformance.yml uses the shared setup API client + github-api-with-retry.js, downloads each registry artifact via Actions artifact APIs, preserves missing vs invalid, uploads the JSON/Markdown report artifact, and gates hard failure only behind enforce_block=true.
• The remaining AC2 evidence gap is runtime, not source code: a live workflow_dispatch run must complete and expose the conformance report artifact contents. I dispatched maint-81-langsmith-fleet-conformance.yml on main with enforce_block=false: https://github.com/stranske/Workflows/actions/runs/26694084113. It was in progress at 2026-05-30T20:26:10Z.

Next closer action: inspect run 26694084113 when complete; if it produced the report artifact distinguishing missing vs invalid, post artifact evidence and close #2184. If the run fails for a bounded workflow defect, fix that PR-local workflow/helper issue.

[stranske]

Closer verifier-disposition completion for PR #2192:

Live dispatch evidence is now present. maint-81-langsmith-fleet-conformance.yml run 26694084113 completed successfully on main: https://github.com/stranske/Workflows/actions/runs/26694084113.

Downloaded artifact langsmith-fleet-conformance-26694084113 contains both JSON and Markdown reports. The JSON report has total_registry_entries: 8 and status counts missing: 8, invalid: 0, stale: 0, valid: 0; each registered repo row reports its current honest missing state (no uploaded artifact yet), and the report schema retains invalid as a distinct status/count for uploaded malformed artifacts. The merged test test_conformance_report_marks_uploaded_malformed_pa_artifact_invalid covers the PA malformed upload path and asserts that such an artifact is classified as invalid with first_error, not silently downgraded to missing.

This satisfies the remaining AC2 verifier concern: the live dispatch run reports every registered repo status and the report/artifact format distinguishes missing artifacts from invalid uploaded artifacts. No follow-up debt remains.

[stranske]

Closing after durable verifier disposition: PR #2192 is merged with verify:compare, verifier CONCERNS were audited against current main, live dispatch run 26694084113 completed successfully, and the conformance report artifact provides the required missing/invalid status evidence.