Uh oh!
There was an error while loading. Please reload this page.
Add verbose Learning Mode diagnostics - #1083
Conversation
* Unify audit with capture denial routing Route wxc-exec --audit through captureDenials so native PSEC/V2 remains preferred and guarded WPR provides compatible analysis and ETL retention. Reuse canonical denials for adjusted policy generation and support retainEtl consistently across both capture providers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Document unified audit architecture Update repository guidance for captureDenials-backed audit routing and guarded retainEtl parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Address audit capture review findings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Filter guarded WPR traces by process Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Document signed PLM developer loop Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Document local PLM signing workflow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Link Windows SDK signing tools Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Address retained trace review findings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Keep interactive traces elevated Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Add Data Loop denial diagnostics Emit a bounded, username-redacted Data Loop JSON sibling containing canonical denial occurrences and classified decoder outcomes. Keep paired output transactional across native, guarded, and audit capture paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 * Preserve Learning Mode Data Loop exclusions Keep excluded namespace evidence uniquely identifiable, retain guarded-trace unknown schemas and brokered capability events, and emit canonical/Data Loop outputs as a transactional pair. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Harden guarded decoding and paired rollback Use targeted broker PID decoding without aborting on unrelated malformed payloads, and make rollback ownership-aware, portable, and no-clobber. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Close Learning Mode review gaps Preserve malformed scoped capability events, retain uncached TDH schemas after the cache cap, and make rollback recovery no-clobber and deletion tolerant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Verify promoted output contents during rollback Detect in-place overwrites before deleting a promoted output and directly exercise TDH short-circuit and cache-capacity behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Complete paired output rollback after cleanup errors Restore backups even when quarantined output cleanup fails, preserve both errors, and hash promoted files from a known offset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Preserve workload PID on partial capability decode Recover the brokered payload ProcessId when full event decoding fails after that property, and drop only events that remain unscopable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Preserve rollback diagnostics and PID fallback coverage Surface quarantine restoration failures with accurate rollback context and codify malformed capability PID fallback scoping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Add path context to paired rollback errors Identify backup, quarantine, and destination paths in cleanup and restoration failures while preserving strict error propagation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf * Redact complete file paths from Data Loop output Keep canonical denial resources actionable while replacing file paths in the diagnostic sibling JSON with <REDACTED>. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Make audit copy fallback race-safe Acquire fallback destinations with create_new and clean up only files owned by the current relocation attempt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Redact path-suffixed Data Loop properties Treat AppPath, ApplicationPath, and other path-suffixed fields as complete file paths before diagnostic bounding or hashing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Make audit artifact relocation transactional Relocate canonical and Data Loop JSON as an ownership-verified pair and stage ETL copy fallbacks through private no-clobber temporary files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Address adversarial review findings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Redact absolute paths for non-file events Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Redact NT filesystem paths in diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Preserve signatures when size fit fails Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Distinguish DOS paths from device identifiers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Decode additional learning mode denial shapes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Rename Data Loop to verbose logging Replace the public and internal Data Loop terminology with verbose logging, including the serialized analysis field and sibling artifact filename. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Use verbose logging terminology Replace Data Loop names in APIs, transport fields, artifact paths, diagnostics, tests, and documentation with verbose logging terminology. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Align fixture event accounting Route decoded test fixtures directly through the production handler so scope filtering and processing limits use the same single accounting path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Harden verbose output relocation Avoid per-run create-new lock artifacts, preserve committed relocation metadata while surfacing source-cleanup warnings, classify malformed event 28 records by schema name, and update the documented output architecture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Address Learning Mode review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Harden output backup ownership Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab --------- Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216 Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab
* Keep non-actionable denials in verbose output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Address actionable output review comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab * Fail closed on unknown registry access Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab --------- Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab
Emit sanitized verbose capture artifacts as bounded, independently parseable MXC.Verbose chunks across executor and SDK completion paths. Version the consent disclosure and document the reconstruction and privacy contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab
Propagate EventWriteTransfer status codes through the verbose telemetry path so rejected chunks produce a bounded local diagnostic instead of failing silently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9e32b4c9-eb47-4231-91de-0da5fde17ab7
Use MXC.VerboseDenials as the ETW identity so the event's Learning Mode denial payload is clear to telemetry consumers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9e32b4c9-eb47-4231-91de-0da5fde17ab7
Apply the privacy-reviewed Microsoft Products disclosure and bump the consent resource version so existing grants require renewed consent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ba4f8739-616b-4b42-8eba-24a762602400
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
The SDK path regresses non-blocking polling and can suppress or silently discard verbose telemetry failures.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds bounded, sanitized Learning Mode diagnostics, publishes them beside actionable denials, and transports them through consent-gated telemetry. It also updates telemetry consent wording and renews consent via resource version 3.
Changes:
- Adds verbose denial aggregation, paired artifact handling, and ETW chunking.
- Extends Windows ETL decoding, filtering, and capture finalization.
- Updates consent resources, SDK documentation, and tests.
File summaries
| File | Description |
|---|---|
tests/scripts/run_telemetry_etw_smoke_test.ps1 | Updates consent expectations. |
tests/scripts/run_telemetry_consent_smoke_test.ps1 | Updates consent smoke assertions. |
src/testing/wxc_e2e_tests/tests/e2e_telemetry_etw.rs | Seeds consent version 3. |
src/mxc_telemetry/src/lib.rs | Adds the verbose-denials ETW event. |
src/host/plm/src/stop.rs | Handles verbose-output path collisions. |
src/host/plm/src/log.rs | Updates analysis fixture. |
src/host/plm/src/elevated.rs | Updates actionable terminology. |
src/host/plm/src/analysis.rs | Writes paired denial artifacts. |
src/host/plm/readme.md | Updates PLM guidance. |
src/core/wxc/src/main.rs | Emits verbose telemetry and cleanup warnings. |
src/core/wxc/src/audit.rs | Relocates and validates artifact pairs. |
src/core/wxc_common/src/telemetry/mod.rs | Adds authorized verbose emission. |
src/core/wxc_common/src/telemetry/events.rs | Defines verbose telemetry records. |
src/core/wxc_common/src/telemetry/consent.rs | Uses the canonical resource version. |
src/core/wxc_common/src/telemetry/consent_prompt.rs | Updates consent prompt tests. |
src/core/wxc_common/resources/telemetry/consent/en-US.json | Introduces consent resource version 3. |
src/core/mxc_engine/src/verbose_telemetry.rs | Validates, chunks, and emits verbose documents. |
src/core/mxc_engine/src/run.rs | Adds audit runner resolution. |
src/core/mxc_engine/src/lib.rs | Integrates verbose SDK telemetry. |
src/core/mxc_engine/Cargo.toml | Adds hashing and randomness dependencies. |
src/core/learning_mode_core/src/verbose_logging.rs | Defines bounded verbose diagnostics. |
src/core/learning_mode_core/src/model.rs | Makes classifications orderable. |
src/core/learning_mode_core/src/lib.rs | Exports verbose and paired-output APIs. |
src/core/learning_mode_core/src/analyze.rs | Adds bounded verbose results. |
src/core/learning_mode_core/Cargo.toml | Adds output-management dependencies. |
src/Cargo.toml | Adds the shared same-file dependency. |
src/Cargo.lock | Locks new dependencies. |
src/backends/learning_mode/windows/src/ui.rs | Updates actionable terminology. |
src/backends/learning_mode/windows/src/tdh_decode.rs | Adds typed and partial property decoding. |
src/backends/learning_mode/windows/src/etl_filter.rs | Preserves known-provider events safely. |
src/backends/learning_mode/windows/src/capability_dacl.rs | Adds sanitized verbose metadata. |
src/backends/learning_mode/windows/Cargo.toml | Adds hashing support. |
src/backends/appcontainer/common/src/guarded_capture.rs | Updates paired-output terminology. |
src/backends/appcontainer/common/src/capture_output.rs | Publishes actionable and verbose artifacts together. |
sdk/node/tests/unit/telemetry.test.ts | Updates consent title fixture. |
sdk/node/tests/unit/default-consent-protocol-runner.test.ts | Updates protocol fixture title. |
sdk/node/README.md | Documents verbose telemetry. |
sdk/dotnet/README.md | Documents verbose telemetry. |
sdk/dotnet/Microsoft.Mxc.Sdk.Tests/MxcTelemetryTests.cs | Updates consent expectations. |
README.md | Documents audit and telemetry changes. |
docs/telemetry/telemetry.md | Documents the new event and inventory. |
docs/telemetry/telemetry-consent-design.md | Updates consent disclosure and version. |
docs/process-container/os-version-support.md | Updates actionable terminology. |
docs/learning-mode/capabilities.md | Documents verbose artifacts and classifications. |
.github/copilot-instructions.md | Updates Learning Mode architecture guidance. |
Review details
Suppressed comments (1)
docs/telemetry/telemetry.md:309
- This second review-status statement still calls the consent wording version 2 and says its privacy review is pending, although the resource is version 3 and the PR identifies the disclosure as privacy-reviewed. Remove the consent wording from the pending list while retaining any genuinely pending verbose-event reviews.
The version 2 consent wording, verbose event inventory, WinExt
classification, retention, access, regional processing, deletion, and
localization/accessibility decisions require explicit privacy review.
- Files reviewed: 47/48 changed files
- Comments generated: 8
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ba4f8739-616b-4b42-8eba-24a762602400
There was a problem hiding this comment.
🟡 Changes recommended
Cross-platform compilation is broken and the telemetry payload can expose unreviewed workload-controlled values.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 48/49 changed files
- Comments generated: 3
- Review effort level: Balanced
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ba4f8739-616b-4b42-8eba-24a762602400
There was a problem hiding this comment.
🔵 Needs a closer look
The telemetry projection can emit unvalidated strings, and some new failure diagnostics remain inaccessible or silently discarded.
Review details
Suppressed comments (4)
Previously missed (3) — in code that hasn't changed since the last review.
src/core/learning_mode_core/src/verbose_logging.rs:155
- When
total_occurrencesis alreadyu64::MAX,record_overflowsaturates it atMAXand the following subtraction reduces it toMAX - 1. Recording another event must never decrease the total; update only the overflow counters here because the total was already incremented before the match.
src/core/mxc-sdk/src/sandbox.rs:178 wait()can both return an error and append the new verbose-telemetry warning (the added engine test constructs exactly this case), but?consumes and dropsselfbefore either warnings or finalized metadata are returned. Consequentlymxc_sdk::runand FFI/SDK callers still cannot diagnose an ETW/artifact failure when backend teardown also fails. Preserve these diagnostics in the error path or return a result type that carries them.
src/core/wxc/src/audit.rs:225- If the JSON pair commits but source cleanup produces warnings, and the subsequent ETL move fails, this
?discards those warnings. The operator then sees only the ETL error and is not told that stale actionable/verbose source files may remain. Include accumulated cleanup warnings in the returned error on this branch.
src/core/mxc_engine/src/verbose_telemetry.rs:153
- Only property values are redacted. This artifact is reopened from a caller-selected path and deserialized without any producer identity/integrity check, so a concurrent replacement can place customer content in
propertiesnames orprovider_guid; both strings survive this projection and are uploaded inmxc.content. Build a closed telemetry projection (for example, derive the GUID from the provider enum and allowlist/drop property names), or bind the reopened file to the artifact produced during teardown.
- Files reviewed: 48/49 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Richie Gomez (richiemsft)
commented
Sep 2, 2026
Copilot's suppressed review also identified three issues in inherited base-PR code that are intentionally not addressed by this stacked PR:
These findings belong to the underlying telemetry work in #822 (or a follow-up from that branch). This PR remains scoped to the new |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ba4f8739-616b-4b42-8eba-24a762602400
📖 Description
Adds bounded, sanitized verbose Learning Mode diagnostics alongside the actionable denial artifact. Non-actionable and unsupported events retain a closed exclusion reason for investigation without changing policy recommendations, and telemetry reports verbose-artifact production and ETW write failures.
Also applies the privacy-reviewed Microsoft Products consent disclosure and bumps the consent resource version to require renewed consent for the material wording change.
This PR is stacked on #822.
🔗 References
Related PRs:
🔍 Validation
cargo test -p wxc_common telemetry::consent_prompt.NET MxcTelemetryTests(51 tests)✅ Checklist
Cargo.lock, thedependency-feed-checkcheck passes (see docs/pull-requests.md)📋 Issue Type
GitHub Actions runs the PR validation build automatically. The ADO pipeline
(
MXC-PR-Build) is the Azure version of the PR pipeline, kept in parity with the GitHubActions build; it runs on merge to
main, and Microsoft reviewers with write access can trigger iton a PR with
/azp run. See docs/pull-requests.md.If the
dependency-feed-checkcheck fails on a new dependency, the crate must be added tothe feed before the PR can pass. See docs/pull-requests.md
for the steps.
Microsoft Reviewers: Open in CodeFlow