Uh oh!
There was an error while loading. Please reload this page.
telemetry: implement deferred crash/cancellation/state-aware lifecycle instrumentation - #624
Conversation
There was a problem hiding this comment.
Pull request overview
Implements the remaining deferred experimental telemetry work across executors and the state-aware lifecycle, adding crash/panic and cancellation instrumentation while tightening the “non-zero exit must surface a diagnostic” contract.
Changes:
- Add lock-free “provider active” tracking and a new
mxc.phasefield to telemetry events. - Add a shared crash/cancellation planning + exactly-once terminal emission guard, plus state-aware lifecycle telemetry emission with per-phase attribution.
- Factor and reuse
emit_backend_error_envelopeso all executors print a machine-readable diagnostic on infrastructure failures before exiting non-zero; update docs accordingly.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/mxc_telemetry/src/lib.rs | Adds lock-free is_active() and threads phase through ETW event emission. |
| src/core/wxc/src/main.rs | Initializes/attributes telemetry on the state-aware path, installs panic hook, emits cancellation telemetry from console handler, and uses shared backend-error envelope emission. |
| src/core/wxc_common/src/telemetry/mod.rs | Adds panic/cancellation/state-aware telemetry plumbing, context attribution, and exactly-once terminal emission guard. |
| src/core/wxc_common/src/telemetry/events.rs | Extends event model with phase and updates error logging signature accordingly. |
| src/core/wxc_common/src/script_runner.rs | Introduces reusable emit_backend_error_envelope helper and basic unit coverage. |
| src/core/wxc_common/src/config_parser.rs | Rejects malformed non-object experimental in state-aware parsing; populates typed experimental.telemetry from raw state-aware block. |
| src/core/mxc_darwin/src/main.rs | Wires telemetry init/emit + panic hook and backend-error envelope emission for parity (no-op telemetry on macOS). |
| src/core/lxc/src/main.rs | Wires panic hook and backend-error envelope emission for parity. |
| docs/telemetry/telemetry.md | Documents state-aware phase telemetry plus crash/cancellation behavior and limitations. |
| docs/schema.md | Reconciles supported-version table with canonical schema version bounds. |
…e instrumentation Follow-up implementing the telemetry issues deferred from the original TraceLogging ETW PR (#493): crash/panic telemetry via a global panic hook (#561), state-aware lifecycle instrumentation with an `mxc.phase` field (#562), cancellation telemetry from the console control handler (#563), non-zero-exit diagnostic parity across the executors (#564), and reconciliation of the `docs/schema.md` supported-version table (#565). All behind the existing `--experimental` + `experimental.telemetry.enabled` gate. The PII invariant is preserved: free-form error text is never emitted (bounded `FailureReason`/`error_type` and numeric exit codes only), and on non-Windows platforms all telemetry functions compile as no-ops. Also unifies the state-aware telemetry config on the typed `experimental.telemetry` field, extracts a global-free crash-emit test seam (making panic/cancellation mapping testable across any backend/phase), and rejects a malformed (non-object) `experimental` block consistently with the one-shot path. Resolves#561Resolves#562Resolves#563Resolves#564Resolves#565 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
67b22aa to
c6a773aCompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…relation Addresses the PR #624 review round. Squashed follow-up to the initial deferred-telemetry commit; folds in the review fixes, the correlation rework, and the adversarial-review hardening. Telemetry (experimental, gated on --experimental + experimental.telemetry): - Crash and cancellation instrumentation with a panic-free, message-free panic hook chained onto the previous hook, wired symmetrically across all three executors (wxc-exec, lxc-exec, mxc-exec-mac). macOS has no ETW sink today, so its emits are structural no-ops kept for parity. - Exactly-once terminal-emit slot made resettable and testable (#[cfg(test)] reset seam), so the panic hook / control handler vs emit_completion race is covered rather than only the inactive no-op path. - Capture-sink / force-active seam over the mxc_telemetry log_execution / log_error calls so emit_panic / emit_cancellation glue (process globals, is_active/already_emitted guards, backend/phase/exit-code) is unit-tested. - Cancellation emit-before-cleanup ordering locked in with a handler-level test (the OS-budget guarantee from the PR description). - Full phase x outcome matrix for plan_state_aware across {provision, start, exec, stop, deprovision} x {success, non-zero guest exit, MxcError}. Cross-phase correlation (Microsoft Correlation Vector, MS-CV): - Replaces the initial per-lifecycle sandbox_id with a full MS-CV (correlation_vector.rs). Seeded at provision, returned in its result, and relayed verbatim by the client into each later phase (emitted under __TlgCV__). The executor is the trust boundary: on each non-provision phase it validates the relayed value and spins a fresh child element off a mutable base, passes an already-frozen vector through unchanged, and reseeds a brand-new base if the relayed value is absent or malformed. - New optional correlationVector wire field (wire.rs), regenerated dev schema and generated SDK wire types, SDK + state-aware plumbing, and telemetry/schema/state-aware docs. - Hardened MS-CV operators/planner per adversarial review: tightened the SPIN_ENTROPY_BYTES compile-time guard (< 4 so the coarse counter still contributes to the 32-bit spun element), documented the deterministic os_random_bytes test seam, and corrected the frozen-vector pass-through wording across wire.rs / schema.md / SDK docs. Backend error surfacing (issue #564): - Shared emit_backend_error_envelope in wxc_common::script_runner so MXC never exits non-zero on an infrastructure failure without first printing a machine-readable {"error":{"code":"backend_error",...}} envelope on stderr. Wired into wxc-exec / lxc-exec / mxc-exec-mac and the state-aware finalize path; covered by a macOS seatbelt integration test (blockedHosts rejection) plus unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 404ca98b-224b-485e-bfe6-c092c5c500d0
RamonArjona4
commented
Jul 13, 2026
Review round addressed — pushed as a single squashed update ( |
| Thread | Ask | How it was addressed |
|---|---|---|
telemetry/mod.rs:493 | "How do I correlate telemetry across phases for a given lifecycle?" | Added a per-lifecycle correlation field, then reworked it into a full Microsoft Correlation Vector (MS-CV) (correlation_vector.rs). Seeded at provision, relayed verbatim by the client into each later phase (emitted under __TlgCV__); the executor validates and spins/reseeds per phase. See the new "Correlating a lifecycle" section in docs/telemetry/telemetry.md. |
telemetry/mod.rs:57 | Exactly-once terminal-emit slot is untestable — HAS_EMITTED is a bare process-global with no reset. | Made the emit slot resettable + testable: added a #[cfg(test)] reset seam clearing HAS_EMITTED/PROCESS_BACKEND/PROCESS_PHASE, plus a test that claims the slot then asserts a second terminal emit is suppressed. |
telemetry/mod.rs:353 | The emit glue (emit_panic/emit_cancellation reading globals, guarding on is_active/already_emitted, writing to the hardcoded sink) has no capture point, so none of it is asserted. | Introduced a capture-sink / force-active seam over the mxc_telemetry::log_execution/log_error calls. Tests now activate emission, invoke emit_panic/emit_cancellation, and assert the captured MXC.Execution + MXC.Error carry the right backend/phase/exit-code (101 / 130). |
wxc/main.rs:514 | Cancellation "emit before cleanup" ordering is a real guarantee but nothing locks it in. | Added a handler-level test asserting emit-before-cleanup ordering (built on the sink seam above), so moving emit_cancellation() below the drain/cleanup now fails a test. |
telemetry/mod.rs:692 | Phase coverage incomplete — stop/deprovision untested, no success/guest-exit/dispatch-error matrix. | Added a table-driven plan_state_aware matrix over {provision, start, exec, stop, deprovision} × {success, non-zero guest exit, MxcError}. |
Also in this update
- MS-CV hardening from an internal adversarial-review pass: tightened the
SPIN_ENTROPY_BYTEScompile-time guard (< 4, so the coarse-time counter still contributes to the 32-bit spun element), documented the deterministicos_random_bytestest seam, and corrected the frozen-vector pass-through wording acrosswire.rs/schema.md/ SDK docs. Both codegen CI gates (dev schema + generated SDK wire types) pass. - Backend error surfacing (Ensure MXC never exits non-zero without first printing an error message #564): shared
emit_backend_error_envelopeinwxc_common::script_runnerso MXC never exits non-zero on an infrastructure failure without first printing a machine-readable{"error":{"code":"backend_error",…}}envelope on stderr — wired into all three executors and the state-aware finalize path, covered by a macOS seatbelt integration test plus unit tests. Flagging this explicitly since it is adjacent-but-distinct from the telemetry work; happy to split it into its own PR if you'd prefer.
cargo fmt/clippy clean; wxc_common + wxc tests green (446 + 25).
Resolves 4 merge conflicts after main advanced 17 commits (notably the mxc_engine refactor that centralized one-shot + state-aware dispatch): - wxc/src/main.rs: keep the state-aware telemetry wrapper in run_state_aware_main but dispatch via mxc_engine::run_state_aware, dropping the now-duplicated local dispatch_state_aware_request; keep the tested StateAwareExit / finalize_state_aware_outcome helpers. - mxc_darwin/src/main.rs: keep the Seatbelt telemetry init + panic hook, drop the orphaned Runner::new(SeatbeltScriptRunner) (dispatch is now mxc_engine::run). - docs/schema.md: adopt canonical min schema version 0.6.0-alpha (schemas/schema-version.json) over the branch's stale 0.4. - state-aware overview doc: adopt sdk/node/ paths, correct docs/schema.md. Verified: cargo build -p wxc, cargo clippy --workspace --all-targets -D warnings, cargo fmt --check, cargo test -p wxc -p wxc_common (445 passed), and check-schema-versions / check-schema-codegen / check-sdk-types-codegen / validate-configs all pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fddfe62a-90da-425e-868f-fae6958d68c6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fddfe62a-90da-425e-868f-fae6958d68c6
Uh oh!
There was an error while loading. Please reload this page.
Resolve conflict in wxc_common/src/state_aware_dispatch.rs: register both the `lxc` (this PR) and `wsb` (upstream microsoft#578) state-aware backend prefixes in backend_from_prefix, and keep both resolve_backend unit tests. Added `correlation_vector: None` to the lxc test to match the ParsedStateAwareRequest field introduced upstream (microsoft#624). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
📖 Description
Follow-up work implementing the telemetry issues deferred from the original
TraceLogging ETW PR (#493). Extends the experimental telemetry surface with
crash/panic, cancellation, and state-aware lifecycle instrumentation, and
tightens the non-zero-exit diagnostic contract, all behind the existing
--experimental+experimental.telemetry.enabledgate.std::panic::set_hookhandler(installed on both the one-shot and state-aware paths) emits a failure
MXC.Execution+MXC.Error(internal_error, exit code101) when anythread panics. No panic message or backtrace text is emitted — only the
bounded category and exit code — and the previous hook is chained so the
default stderr backtrace still prints.
stop/deprovision phases now emit telemetry carrying an
mxc.phasefield.close / shutdown) emits a cancellation event before cleanup.
non-zero exit never occurs without first surfacing a diagnostic; extracted
a shared
emit_backend_error_envelopefor lxc/darwin parity with wxc.docs/schema.mdsupported-versiontable with
schemas/schema-version.json.The change keeps the established PII invariant: free-form error text is never
emitted — only a bounded
FailureReason/error_typeenum and numeric exitcodes. On non-Windows platforms all telemetry functions compile as no-ops.
Includes review-driven hardening from adversarial + security + Copilot-style
reviews: the two telemetry-config parser paths were unified on the typed
experimental.telemetryfield, a global-free crash-emit test seam wasextracted (making panic/cancellation mapping testable across any
backend/phase), and the state-aware parser now rejects a malformed
(non-object)
experimentalblock consistently with the one-shot path.Limitation (documented in
docs/telemetry/telemetry.md): on backends thatrecover panics via
catch_unwind(the LXC runner), the panic hook records the101sentinel and claims the exactly-once terminal-emit slot, so the recoveredcompletion event is suppressed. There,
mxc.exit_code = 101is a"panic occurred" marker rather than the observed process exit code.
🔗 References
Resolves#561
Resolves#562
Resolves#563
Resolves#564
Resolves#565
Related (deferred follow-up, not resolved here):
equivalent), deferred from the field-comment mechanism discussion.
🔍 Validation
Automated (from
src/, toolchain 1.93):cargo fmt --all -- --check— cleancargo clippy --workspace --all-targets -- -D warnings— no warningscargo test --workspace— all pass (adds panic/cancellation/state-awaretelemetry unit tests, pure crash/state-aware event-planner tests, and
state-aware parser tests including malformed/non-object
experimentalrejection)
CI gates run locally (all green):
check-version-sync,check-schema-versions,check-rust-toolchain-synccheck-schema-codegen+check-sdk-types-codegen(no wire-model drift)validate-configs(173 configs validated against the dev schema)Reviews: 7-axis adversarial review (findings fixed), security review (clean),
and an emulated GitHub Copilot PR review (three findings fixed).
✅ Checklist
Cargo.lock, thedependency-feed-checkcheck passes — n/a,Cargo.lockis unchanged📋 Issue Type
Microsoft Reviewers: Open in CodeFlow