Skip to content

feat(ocsf): comprehensive audit telemetry and off-box OTLP export - #3

Open
mattj-monad wants to merge 3 commits into
mainfrom
feat/gateway-ocsf-audit-telemetry
Open

feat(ocsf): comprehensive audit telemetry and off-box OTLP export#3
mattj-monad wants to merge 3 commits into
mainfrom
feat/gateway-ocsf-audit-telemetry

Conversation

@mattj-monad

@mattj-monadmattj-monad commented Aug 22, 2026

Copy link
Copy Markdown

TLDR

OpenShell can run agents safely, but understanding the actions the agent took (or were denied by policy) is limited to a subset of actions and the logs cannot be exported for analysis.. This PR makes every security-relevant action a structured, standard-schema (OCSF) event and streams the logs to any OpenTelemetry collector. It turns OpenShell into an enterprise ready solution with full visibility, which can audited, monitored, and learned from.

Why it matters

  • Understand and improve your agents and models. Every choice an agent takes or has blocked, the commands it runs, the resources it reaches for, the policies and boundaries it hits, where it stalls or retries, all becomes collectable data. This lets teams building on OpenShell see how their agents actually behave in the wild and rapidly improve their own models, prompts, and policies from real usage rather than guesswork, all while providing the security team visibility.
  • Built for enterprise scale. One correlated, standard-schema stream with accountable delivery, designed to run across thousands of sandboxes at once. An efficient push format keeps the per-event cost low so the volume stays manageable at fleet scale.
  • Audit — a provable record of who created, changed, minted, or deleted anything: sandboxes, credentials, policies, tokens, workspaces. It answers "who deleted this sandbox?" and "was this credential rotated by a person or by a job?" — questions that had no data behind them before.
  • Security — authentication failures and boundary violations become first-class alerts.
  • Open standard, no lock-in — everything is OCSF (the security schema OpenShell already uses for sandbox events) carried over OTLP (the transport agent frameworks already speak), so it drops into tooling teams already run.

What we built

  • Control-plane audit (new). 34 state-changing gateway operations that previously emitted nothing now each produce one structured event carrying the authenticated actor and the outcome — across workspaces, providers, credentials, sandboxes, SSH sessions, exec, policy, settings, inference routes, and token issuance. Background work (credential rotation, reconciliation, startup) is attributed to a system actor instead of going unrecorded.
  • Authentication & boundary alerts (new). Auth failures are recorded with mechanism and a safe reason category; cross-boundary attempts dual-emit a high-severity security finding alongside the ordinary denial.
  • Log Export (new). The sandbox's existing OCSF security events and these new control-plane events export together as OTLP logs to any OpenTelemetry collector, correlated per sandbox, on the same endpoint the gateway already uses for traces.
  • Accountable delivery. Bounded buffering with retries on both hops (sandbox→gateway and gateway→collector); anything dropped under load surfaces as a countable gap, so the record is never silently incomplete.
  • Safe by construction. No credential, token, or bearer material ever enters a record. A single master toggle governs the whole trail; it's on by default.

Example

Every event is human-greppable shorthand and full OCSF JSON. A sandbox deletion, for example, renders as:

ENTITY:DELETE [INFO] sandbox "sb-abc123" by user:alice(alice@corp)

...and exports as a complete OCSF Entity Management record carrying the actor, outcome, correlation id, and severity for downstream filtering and alerting.

Scope

Four OCSF event classes (Entity Management, Config State Change, Authentication, Detection Finding), ~34 instrumented handlers, spanning the shared OCSF and OTLP crates, the sandbox supervisor, and the gateway.

Alignment with roadmap

This is the built, tested implementation of OpenShell's own Enterprise Observability direction (NVIDIA#1055) and the issues the maintainers have already broken it into: control-plane audit events (NVIDIA#2911), gateway authentication and boundary findings (NVIDIA#2912), OCSF-over-OTLP export (NVIDIA#2892), and accountable relay delivery (NVIDIA#2642). Sandbox and control plane, one correlated stream, off the box — ready now.

Next steps.

We are working with the OpenShell maintainer team to get the fixes and issues aligned with requests and requirements. Once merged, these contributions should allow enterprise teams to adopt OpenShell with full visibility into the entire projects actions.

Add the delivery substrate that carries gateway and sandbox logs, and
OCSF security events, off the box to any OTLP/gRPC collector, plus the
OCSF v1.7.0 event model they travel in.
- Export gateway and sandbox logs as OTLP log records on the same
endpoint and resource identity as traces, opt-in via
`[openshell.gateway.otlp] export_logs`, with TLS for https collectors.
- Accountable delivery (Tier 2a): a bounded in-memory queue, the
exporter driven directly rather than through the SDK batch processor,
bounded retries, and a `telemetry_gap` record for every dropped line
so loss is never silent.
- OCSF event classes, builders, objects, enums, shorthand and JSONL
formatters, and a tracing-layer bridge; `Actor` gains an optional
user alongside its now-optional process.
- Raw is the default push format: the full OCSF document travels in one
`ocsf.raw` attribute (with `ocsf.severity_id`), several-fold cheaper
per record than the flattened shape; `OPENSHELL_OCSF_PUSH_FORMAT=flat`
opts back. OCSF events export at their own severity.
- Gateway-scoped events (no sandbox in play) export on a gateway lane
with their structured fields as attributes; export-path targets are
excluded so a failing collector cannot amplify its own failure.
- Benchmarks for the fan-in and export paths.
Signed-off-by: clippy <matt@monad.com>
Every state-changing gateway operation now emits one OCSF audit event
carrying the authenticated principal as the actor, the outcome, and a
request-id for correlation. Governed by `[openshell.gateway.audit]`
(master `enabled` on by default, mirrored by env vars and CLI flags with
CLI > env > TOML precedence).
Event classes:
- Entity Management [3004]: workspace, member, provider, profile,
credential, sandbox lifecycle, provider attach/detach, ssh session,
exec and tcp-forward session initiation, and sandbox-token minting.
- Config State Change [5019]: settings (before/after values, credential
keys always redacted), policy replace/merge, draft-chunk decisions,
inference routes, and a companion marker when a gateway interceptor
rewrites a request in flight.
- Authentication [3002]: failures always (mechanism, low-cardinality
reason, peer address; never the credential); per-request successes
behind a toggle.
- Detection Finding [2004]: cross-sandbox access attempts (up-front and
mid-stream) and sandbox-principal admin attempts, High and alert-grade,
dual-emitted beside the domain denial.
Guarantees: outcomes judged from the store (no-op deletes record as
Failure at Low severity so alerting sees them); events emit at the store
commit so a landed change is on record even if a later step fails;
secrets never enter a record; actors come from the session and render as
name(uid); the `ocsf` tracing target is pinned at INFO so a coarse
RUST_LOG cannot silently sever the trail. Background mutations
(credential auto-rotation, compute reconciliation, startup) audit under
`system:<component>` actors.
Signed-off-by: clippy <matt@monad.com>
Add a three-page gateway-audit guide under docs/observability/ (overview
and how-to, the complete toggle reference, and the full event catalog
with an example and use-cases per class), wired into the docs navigation
and cross-linked from the sandbox logging page, the gateway config
reference, and the README. Extend the config reference and Helm chart
with the OTLP export and `[openshell.gateway.audit]` surfaces, update the
architecture logging overview, and record the emit-an-audit-event rule
for new state-changing RPCs in AGENTS.md.
Signed-off-by: clippy <matt@monad.com>
@mattj-monadmattj-monad changed the title Feat/gateway ocsf audit telemetryfeat(ocsf): comprehensive audit telemetry and off-box OTLP exportAug 26, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@mattj-monad