Add verbose Learning Mode denial logging

[Richie Gomez (richiemsft)]

📖 Description

Adds deterministic verbose logging for every successful Learning Mode denial analysis while preserving the canonical policy-output contract and expanding canonical decoding for newly validated ETL shapes.

Each analysis now produces a transactional pair:

• denials.json retains canonical, actionable denials, including complete file paths.
• denials.verbose.json retains a bounded, deduplicated diagnostic superset. Complete file paths are replaced with <REDACTED>; standalone account names and username components in non-file identifiers are also redacted before bounding or hashing.

The verbose logging output includes canonical denial occurrences and closed classifications for outcomes omitted from canonical policy generation, including unusable resource paths, unsupported object types and event schemas, malformed payloads, decoder limits, and unresolved capability events. Stable non-file identifiers such as SIDs, capability names, provider GUIDs, PIDs, and named-object details remain available for diagnosis.

The guarded WPR path now preserves known-provider events through process-scoped selection and replay. Brokered event 28 is scoped using its payload ProcessId, including partial-decode recovery, rather than the broker's ETW header PID.

Native capture and plm stop share a transactional paired-output writer. Both JSON files are staged and synchronized before no-clobber promotion, with ownership- and digest-aware rollback. Audit mode consumes these capture artifacts directly and relocates both the JSON pair and retained ETL through no-clobber staging with identity- and digest-verified source cleanup, keeping metadata truthful without deleting files owned by another process.

Additional ETL findings

Validation across eight new namespace scenarios (16 retained ETLs: block and allow) identified additional canonical decoder shapes:

• Event 28 is schema-dependent. UI-shaped payloads containing Category and Detail now emit canonical UI denials such as resource: "Handles", resourceType: "ui", rather than incorrectly treating the payload's package SID as a capability.
• Named Section, SymbolicLink, and Timer access checks now emit canonical other resources with object-specific access-mask classification.
• The UI Handles scenario represents JOB_OBJECT_UILIMIT_HANDLES, primarily cross-process HWND/USER-object handle access.
• SCM, ETW provider/session, and power-policy scenario-specific checks still expose empty ObjectType and ObjectName. They remain bounded verbose logging diagnostics because the ETL provides no stable canonical resource identifier.

The updated decoder increased canonical coverage in 15 of the 16 supplied ETLs. The remaining ETW Session block trace was an early process-initialization failure and contained none of the newly supported named shapes.

🔗 References

Builds on the Learning Mode capture hardening from #847.
This is for issue #828

🔍 Validation

• learning_mode_core: 43 tests passed
• learning_mode_windows: 224 tests passed
• appcontainer_common: 247 tests passed
• plm: 226 tests passed, 1 ignored
• wxc: 40 tests passed
• Clippy passed with warnings denied for learning_mode_core, learning_mode_windows, and plm
• Release wxc-exec.exe and plm.exe build
• Guarded-WPR VM validation with canonical and typed verbose logging groups, zero overflow, and no username leakage
• Production lm_analyze decoder run against all 16 supplied retained ETLs
• Confirmed canonical ui/Handles, Timer, Section, and SymbolicLink output; confirmed event 28 UI no longer emits a bogus package-SID capability
• Fixture accounting now follows the production handler exactly; out-of-scope events do not consume the processing budget
• Create-new output pairs no longer create persistent per-run lock files; relocation publishes committed destinations and surfaces source-cleanup warnings
• Malformed event 28 diagnostics classify UI/capability resources only when the schema name identifies the payload shape
• GitHub Actions build, lint, integration, CodeQL, CLA, and dependency-feed checks passed before the latest ETL decoder commit; replacement checks run automatically

✅ Checklist

• Signed the Contributor License Agreement
• Linked to an issue
• Updated documentation (if applicable)
• Updated Copilot instructions (if build, architecture, or conventions changed)
• If this PR changes Cargo.lock, the dependency-feed-check check passes (see docs/pull-requests.md)

📋 Issue Type

• Bug fix
• Feature
• Task

---

GitHub Actions runs the PR validation build automatically. The ADO pipeline
(MXC-PR-Build) is the Azure version of the PR pipeline, kept in parity with the GitHub
Actions build; it runs on merge to main, and Microsoft reviewers with write access can trigger it
on a PR with /azp run. See docs/pull-requests.md.

If the dependency-feed-check check fails on a new dependency, the crate must be added to
the feed before the PR can pass. See docs/pull-requests.md
for the steps.

Microsoft Reviewers: Open in CodeFlow

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds bounded, redacted Data Loop diagnostics alongside Learning Mode denial output and integrates them across native capture, guarded WPR, and audit workflows.

Changes:

• Adds the Data Loop model, aggregation, sanitization, and paired output.
• Adds process-scoped guarded-WPR ETL relogging and retention.
• Reworks --audit to consume captureDenials artifacts directly.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
.github/copilot-instructions.md	Updates Learning Mode architecture guidance.
README.md	Revises audit-mode documentation.
docs/learning-mode/capabilities.md	Documents Data Loop and retained ETL behavior.
src/Cargo.lock	Records the new dependency edge.
src/backends/appcontainer/common/src/appcontainer_runner.rs	Supports guarded ETL transfer.
src/backends/appcontainer/common/src/base_container_runner.rs	Integrates retained guarded captures.
src/backends/appcontainer/common/src/capture_output.rs	Adds paired denial/Data Loop output.
src/backends/appcontainer/common/src/guarded_capture.rs	Extends guarded-capture interfaces.
src/backends/learning_mode/windows/src/capability_dacl.rs	Adds diagnostic metadata to recovered denials.
src/backends/learning_mode/windows/src/etl_decode.rs	Aggregates Data Loop outcomes and selects relog events.
src/backends/learning_mode/windows/src/etl_filter.rs	Adds process-scoped ETL relogging.
src/backends/learning_mode/windows/src/extractors.rs	Adds sanitization and diagnostic classification.
src/backends/learning_mode/windows/src/lib.rs	Exports ETL filtering.
src/backends/learning_mode/windows/src/tdh_decode.rs	Categorizes recoverable decoding failures.
src/core/learning_mode_core/src/analyze.rs	Carries and bounds Data Loop results.
src/core/learning_mode_core/src/data_loop.rs	Defines the Data Loop model and writer.
src/core/learning_mode_core/src/lib.rs	Exports Data Loop APIs.
src/core/learning_mode_core/src/model.rs	Makes classifications sortable.
src/core/mxc_engine/src/guarded_capture.rs	Implements guarded trace transfer.
src/core/mxc_engine/src/lib.rs	Exports audit runner resolution.
src/core/mxc_engine/src/run.rs	Adds audit-specific resolver entry point.
src/core/wxc/Cargo.toml	Adds Learning Mode core dependency.
src/core/wxc/src/audit.rs	Relocates and post-processes capture artifacts.
src/core/wxc/src/main.rs	Routes audit through captureDenials.
src/host/plm/readme.md	Documents guarded analysis and validation.
src/host/plm/src/analysis.rs	Returns canonical documents after writing.
src/host/plm/src/elevated.rs	Adds filtered analysis/trace transfer protocol.
src/host/plm/src/log.rs	Consumes guardian-returned analysis directly.
src/host/plm/src/secure_scratch.rs	Manages filtered ETL scratch storage.
src/host/plm/src/stop.rs	Extracts canonical-denial post-processing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/core/learning_mode_core/src/data_loop.rs:184

• Once the signature set contains only canonical groups, every new distinct canonical event calls evict_one_noncanonical_group, which scans all 4,096 entries before overflowing it. At the 1,000,000-event bound this can perform billions of comparisons, turning an intentionally bounded diagnostic path into a CPU bottleneck. Track whether any noncanonical group exists (or partition/index the two classes) so the full scan is skipped when eviction cannot succeed.

 while self.signatures.len() >= MAX_DATA_LOOP_GROUPS
|| retained_bytes.saturating_add(serialized_len) > max_bytes
{
if !signature.reason.is_canonical_denial()
|| !self.evict_one_noncanonical_group(Some(retained_bytes))
{

src/host/plm/src/elevated.rs:2341

• This helper is now also used for guarded captureDenials retention, so create_dir_all makes a missing outputPath parent succeed only when guarded WPR plus retainEtl is selected. Native and non-retained captures still fail, and the documented contract says the parent must already exist. Preflight the configured output directory consistently before capture, rather than creating it as a side effect of ETL transfer.
  src/core/learning_mode_core/src/paired_output.rs:56
• Every capture creates a unique .pair.lock file, but Drop only unlocks it and never removes it. Because capture output names include a per-run ID, these lock files are never reused and accumulate indefinitely in the configured output directory (or the system temp directory for managed outputs). Use an ephemeral/delete-on-close lock strategy, or add race-safe cleanup after the transaction.

impl Drop for OutputPairLock {
fn drop(&mut self) {
let _ = std::fs::File::unlock(&self.file);
}

.github/copilot-instructions.md:216

• This architecture entry still says the output layer writes a single JSON document, but this PR makes learning_mode_core own the DataLoopDocument and paired-output transaction as well. Leaving the repository guidance singular will mislead future changes about the required canonical/Data Loop pair; update this entry to describe both artifacts and their paired emission contract.

- `learning_mode_core` is the **cross-platform learning-mode / captureDenials model + output emitter**: `DeniedResource` (+ `ResourceType`/`AccessType`), `DenialSummary`, the `DenialAnalyzer` decode trait, and `emit` — which writes the on-disk denials deliverable as a **single JSON document** `{ "denials": [...], "summary": {...} }` (`write_document` / `DenialsDocument`) and defines the serializable `DenialsOutputPointer`. It carries no OS-specific code (must not depend on any `backends/*` crate); the Windows ETL decoder implementing `DenialAnalyzer` lives in `backends/learning_mode/windows`. When `processContainer.captureDenials` is set, native PSEC/V2 seals and decodes a managed ETL locally, while guarded WPR relogs its host-wide source into a process-scoped ETL before analysis; both routes write the same canonical JSON through shared output plumbing and return neutral `wxc_common` metadata. Explicit `retainEtl` preserves the native sealed trace or the guarded process-scoped relogged trace after a terminal wait; abandonment discards it. `wxc-exec` serializes the metadata as the one-line stderr pointer at the CLI boundary; Rust/C#/FFI callers receive it programmatically. Each denial's `resource` field holds the file path or the AppContainer capability name; capability denials resolve their capability SID to a friendly name via `backends/learning_mode/windows`'s `capability_names` (well-known `S-1-15-3-…` SID → policy name; custom hashed SIDs fall back to the SID string).

[Darren Hoehna (dhoehna)]

Canonical

Does copilot really need to use this every where?

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

docs/learning-mode/capabilities.md:250

• This file-denial example does not match the emitted schema. For a decoded access-check containing ObjectName, record_raw_denial_outcome retains that property (redacted) and deliberately does not synthesize resource; consumers following this example will look for a field that is absent from normal file signatures. Use ObjectName here instead.

 "properties": [
["PackageSid", "S-1-15-3-1"],
["resource", "<REDACTED>"]
]

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/backends/learning_mode/windows/src/extractors.rs:152

• Event 28 is being classified by the presence of both Category and Detail, not by its TDH schema. A UI-shaped LearningModeViolation event with a malformed/missing Detail therefore falls through to the capability extractor and is logged as capability/notActionable (or unresolvedCapability) instead of a UI payload error. This also contradicts the PR's stated rule that malformed event-28 diagnostics use the schema name to choose UI versus capability. Preserve the schema event name in DecodedEventParts and use it for event-28 routing; property presence can then remain validation within the chosen extractor.

 CAPABILITY_DENIAL_EVENT_ID if is_ui_violation_schema(parts) => {
build_denial_from_learning_mode(parts, pid, filetime, provider)
}
CAPABILITY_DENIAL_EVENT_ID => build_denial_from_capability(parts, pid, filetime, provider),

src/backends/learning_mode/windows/src/extractors.rs:649

• Adding SymbolicLink as ResourceType::Other sends it through Accumulator::add_raw_denial's existing non-file normalization (etl_decode.rs:254-256). For the covered object name \??\C:, path_norm::to_user_visible strips the object-manager prefix and the canonical output becomes C:, even though the documentation promises the raw identifier for other resources. This loses namespace information and can merge distinct identifiers. Keep object-manager names raw (or restrict user-visible path normalization to ResourceType::File) before emitting and deduplicating these new denial types.

 let resource_type = match object_type_str {
"File" => ResourceType::File,
"Key" => ResourceType::Other,
"Section" | "SymbolicLink" | "Timer" => ResourceType::Other,
// A present-but-empty object type is a brokered-capability check.
"" => ResourceType::Capability,
_ => return Err(VerboseLoggingExclusionReason::UnsupportedObjectType),

[Richie Gomez (richiemsft)]

Branden Bonaby (@bbonaby), could you take a look?

[Richie Gomez (richiemsft)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

Richie fixed issues. Unblocking to allow for this update to go in.

[Branden Bonaby (bbonaby)]

issue (non-blocking): Could we preserve a no-clobber rename-first fast path here when the platform supports it, then fall back to this copy-and-verify flow? Retained ETLs can be large, and this now copies the whole trace and hashes it multiple times even for a same-volume move that used to be an O(1) rename.

[Branden Bonaby (bbonaby)]

Richie Gomez (@richiemsft) lets follow up via GH issues

[Branden Bonaby (bbonaby)]

issue (non-blocking): Could we remove this second recording path or make it a thin wrapper around record_with_byte_budget? Production only uses the byte-bounded method, while this public version duplicates the state machine with weaker limits, which makes it easy for a future caller to bypass the WPR frame-size guard by accident.

[Branden Bonaby (bbonaby)]

Richie Gomez (@richiemsft) lets follow up via GH issues