feat(ocsf): configurable schema version for SIEM backward compatibility

[zanetworker]

Summary

Add a gateway-configurable OCSF schema version target that downgrades JSONL output for SIEMs that only support older schema versions. AWS Security Lake requires v1.1.0, Splunk CIM Add-On targets v1.1-v1.3, CrowdStrike FDR uses v1.5.0. OpenShell's internal model stays at the current version; only the JSONL serialization is transformed.

Related Issue

Closes#2662

Approach

A downgrade filter in the JSONL layer runs after event serialization and before writing. Based on a field-by-field diff across v1.1.0, v1.3.0, and v1.7.0 (analysis), the filter is small:

1. Strip profile-gated fields: ai_model, container, observation_point_id
2. Remove unknown profiles from metadata.profiles: ai_operation, container
3. Rewrite metadata.version to the target version

No field renames needed. The core event structure (class UIDs, activity IDs, HTTP/network fields) is identical across all versions. The differences are all profile-gated additions.

The shorthand log output is unaffected (human-readable, not consumed by SIEMs).

Gateway Configuration

[openshell.gateway]
ocsf_schema_version = "1.1"

The setting flows to sandboxes via the settings bundle (same mechanism as ocsf_json_enabled). Empty or unset means no downgrade.

Changes

openshell-ocsf crate:

• format/downgrade.rs — downgrade_event() function that strips fields and profiles based on target version
• tracing_layers/jsonl_layer.rs — with_target_version() method; applies downgrade before writing

openshell-sandbox crate:

• main.rs — creates shared ocsf_schema_version mutex, passes to JSONL layer
• lib.rs — apply_ocsf_schema_version_setting() reads the gateway setting and updates the shared version

SIEM Compatibility Matrix

SIEM	OCSF Version	Downgrade Needed
AWS Security Lake	v1.1.0	Yes
Splunk CIM Add-On	v1.1-v1.3	Yes
CrowdStrike FDR	v1.5.0	Yes (strips v1.7+ fields)
Datadog Cloud SIEM	v1.5.0 (selectable)	Depends on config

Testing

• cargo test -p openshell-ocsf — 134 passed (7 new downgrade tests)
• cargo clippy --all-targets -- -D warnings — clean
• cargo check -p openshell-sandbox — compiles with settings wiring

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Unit tests added

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[zanetworker]

E2E Verification: Splunk CIM mapping with v1.1 downgraded events

Tested the full pipeline on a live cluster with Splunk Enterprise:

1. Gateway setting: ocsf_schema_version = "1.1"
2. Sandbox emits events internally at v1.7, JSONL layer downgrades to v1.1
3. Events sent to Splunk HEC with sourcetype=ocsf
4. OCSF-CIM Add-On maps events to standard CIM data models

Splunk CIM query:sourcetype=ocsf tag=network tag=communicate | table _time dest dvc vendor_product metadata.version

Result: 13 events mapped to CIM Network Traffic data model with metadata.version = 1.1. CIM fields (dest, dvc, vendor_product) populated automatically via the OCSF-CIM Add-On's eventtypes and field aliases. No custom dashboards or SPL required.

This proves the "plug OpenShell into your existing SIEM and it just works" story for customers running Splunk with the OCSF-CIM Add-On on v1.1 schemas.

[zanetworker]

Related:#2664 adds AI inference events via the ai_operation profile on ApiActivity [6003] and bumps the schema to v1.8.0. The downgrade filter in this PR strips those profile fields when targeting v1.1/v1.3 for older SIEMs. The two PRs are independent and can be merged in either order.

[johntmyers]

@zanetworker does this configuration need to be addressed in the docs?

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid implementation of accepted issue #2662; the author also supplied live Splunk v1.1 integration evidence.
Head SHA: 8fdaeece2ce7396ade50850d4de6765a01c01985
Base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Merge base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Patch ID: 99ebca72e7706c76032c9ed2d3289e7b3ea8930c
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Thanks @zanetworker, I checked the live Splunk verification and the note that #2664 is independent; those support the feature's validation. @johntmyers's question about documentation is also well-founded: the existing published OCSF export contract becomes inaccurate at this head.

Blocking findings:

• GATOR-8fdaeece-01: Unsupported and malformed target versions are accepted; see the inline finding.
• GATOR-8fdaeece-02 — Warning: Published OCSF export guidance must accurately document selectable output versions and the supported configuration workflow.
  • Prerequisite: An operator follows /observability/ocsf-json-export to configure or integrate JSON export.
  • Entry point → sink: Published OCSF export docs and openshell settings → schema-aware SIEM ingestion of the emitted JSONL.
  • Base → head: The page accurately promised complete v1.7.0 objects before schema selection existed; this head adds field-stripping, schema-selectable output while the page still promises every record is complete v1.7.0 and gives no selection workflow.
  • Impact: The primary integration page materially misstates the emitted contract and leaves operators unable to discover or correctly configure the feature.
  • Reproducer: Follow docs/observability/ocsf-json-export.mdx, then set ocsf_schema_version; downgraded records contradict its v1.7.0 guarantee, and the page has no 1.1/1.3 instructions.
  • PR ownership: The contradiction is created by this PR's new public setting; the relevant docs page already exists but is unchanged, so this blocker cannot be anchored to a docs diff line. The unchanged contract is at docs/observability/ocsf-json-export.mdx:42 and :164; .agents/skills/openshell-cli/SKILL.md:616 also documents the neighboring settings workflow but omits this key.
  • Requested change: Update the export page with global/per-sandbox commands, exact supported values, default/unset behavior, hot-reload timing, metadata rewriting, and omitted fields; update the CLI skill guidance; reference the new screenshot or remove it.

Carried findings:

• None

Non-blocking suggestions:

• None

Docs: Missing for a direct user-facing gateway setting; the current export page is materially false.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid implementation of accepted issue #2662; the author also supplied live Splunk v1.1 integration evidence.
Head SHA: da10094fda901b31d2ead35f8e6c243b921ac291
Base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Merge base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Patch ID: c14374c353d8b1657f1c4dcdcf47a657a40a6bf9
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: 8fdaeece2ce7396ade50850d4de6765a01c01985
Review budget exhausted: no
Maintainer decision required: no

Thanks @zanetworker. I checked the validation and documentation update in your latest commit. The allow-list and tests resolve GATOR-8fdaeece-01, but the published configuration guidance still needs one consistency pass.

Blocking findings:

• GATOR-8fdaeece-02 — Warning (carried): Published OCSF export guidance must document only configuration paths and target versions that the runtime accepts.
  • Prerequisite: An operator follows the new OCSF export documentation to configure the downgrade through gateway.toml or for a v1.5 SIEM integration.
  • Entry point → sink: the documented gateway --config / OPENSHELL_GATEWAY_CONFIG path or openshell settings set → gateway TOML deserialization or RegisteredSetting::validate_string_value.
  • Base → head: The previous head omitted the setting. This head documents [openshell.gateway].ocsf_schema_version, but the gateway file schema has no such field and rejects it; it also recommends 1.5 for CrowdStrike and Datadog even though the new allow-list accepts only empty, 1.1, and 1.3.
  • Impact: Following the TOML example deterministically prevents gateway startup, while following either v1.5 recommendation produces a settings validation error and cannot configure the advertised integration.
  • Reproducer: Start the gateway with [openshell.gateway] containing ocsf_schema_version = "1.1"; parsing rejects the unknown field. Separately run openshell settings set --global --key ocsf_schema_version --value "1.5"; the new allow-list rejects it.
  • PR ownership: This PR introduces the setting, and the latest delta adds these incompatible examples while resolving the existing docs obligation.
  • Locations: docs/observability/ocsf-json-export.mdx:160, :181, :182, and :196; docs/reference/gateway-config.mdx:81.
  • Requested change: Remove the gateway.toml example and gateway-config reference unless that entry point is implemented. Remove or clearly mark the v1.5 workflows as unsupported, and state that selectable downgrade targets are 1.1 and 1.3 through global or per-sandbox settings.

Carried findings:

• GATOR-8fdaeece-02: still open as described above.

Docs: Present, but currently advertises unsupported configuration paths and values.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid implementation of accepted issue #2662; the author also supplied live Splunk v1.1 integration evidence.
Head SHA: 12d577fe83d0b7783aa990b3681263c15d781acb
Base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Merge base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Patch ID: c080fb79f09d08592568aab8d1fc92191a44020c
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: da10094fda901b31d2ead35f8e6c243b921ac291
Review budget exhausted: no
Maintainer decision required: no

Thanks @zanetworker. I checked your latest documentation-only delta against the runtime setting allow-list and configuration path. It removes the unsupported gateway.toml field, removes the gateway configuration reference, and marks the unsupported v1.5 targets accurately. That resolves GATOR-8fdaeece-02 and addresses @johntmyers's documentation question.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None. GATOR-8fdaeece-01 and GATOR-8fdaeece-02 are resolved by the latest patch.

Docs: Fern documentation and the related CLI skill now describe the supported global/per-sandbox settings workflow, supported values, default behavior, and hot-reload timing. No navigation change is needed for the existing page.

Next state: gator:in-review pending required E2E workflow dispatch; advance to gator:watch-pipeline only after the current-head workflow is queued.

[github-actions]

Label test:e2e applied, but pull-request/2717 does not exist yet. A maintainer needs to comment /ok to test 12d577fe83d0b7783aa990b3681263c15d781acb to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[johntmyers]

/ok to test 12d577f

[lbelyaev]

On @johntmyers 's docs question — the thing most worth calling out is that the downgrade is lossy in one specific way. downgrade_event strips ai_model and the ai_operation profile, then rewrites metadata.version, so a downgraded line is byte-for-byte indistinguishable from an event where no model was involved. For additive profile fields that's harmless, but ai_model is the attribution field — for an agent runtime it's close to the one thing the audit stream exists to record — and your Splunk run shows the downgraded JSONL is exactly what the SIEM ingests as the record, with no full-fidelity line beside it.

Would it be worth having a lossy downgrade leave a breadcrumb — stash the elided fields (or just downgraded_from: "1.7.0") under unmapped, which older schemas tolerate and SIEMs ignore — so the record stays honest about what it dropped? The stripping itself is unavoidable for older consumers; the question is only whether the emitted event should admit it happened, or whether the JSONL is explicitly a lossy export with fidelity kept elsewhere. Either way it's worth a line in the docs so an operator on v1.1 knows AI attribution won't be in that stream.

[github-actions]

Label test:e2e applied for 12d577f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

On @johntmyers 's docs question — the thing most worth calling out is that the downgrade is lossy in one specific way. downgrade_event strips ai_model and the ai_operation profile, then rewrites metadata.version, so a downgraded line is byte-for-byte indistinguishable from an event where no model was involved. For additive profile fields that's harmless, but ai_model is the attribution field — for an agent runtime it's close to the one thing the audit stream exists to record — and your Splunk run shows the downgraded JSONL is exactly what the SIEM ingests as the record, with no full-fidelity line beside it.

Would it be worth having a lossy downgrade leave a breadcrumb — stash the elided fields (or just downgraded_from: "1.7.0") under unmapped, which older schemas tolerate and SIEMs ignore — so the record stays honest about what it dropped? The stripping itself is unavoidable for older consumers; the question is only whether the emitted event should admit it happened, or whether the JSONL is explicitly a lossy export with fidelity kept elsewhere. Either way it's worth a line in the docs so an operator on v1.1 knows AI attribution won't be in that stream.

I'm a bit ambivalent on this. Curious what @zanetworker thinks as he's closest to the use case. I usually default to "doing less" when able. Operators working with OCSF should know what version they are on and also understand there are other versions out there that they are missing the benefits from. IMHO we could call this out in the actual docs pages "OpenShell supports up to version 1.8 in order support newer AI properties, if you need this data you should upgrade." or something to that effect.

[lbelyaev]

On @johntmyers 's docs question — the thing most worth calling out is that the downgrade is lossy in one specific way. downgrade_event strips ai_model and the ai_operation profile, then rewrites metadata.version, so a downgraded line is byte-for-byte indistinguishable from an event where no model was involved. For additive profile fields that's harmless, but ai_model is the attribution field — for an agent runtime it's close to the one thing the audit stream exists to record — and your Splunk run shows the downgraded JSONL is exactly what the SIEM ingests as the record, with no full-fidelity line beside it.
Would it be worth having a lossy downgrade leave a breadcrumb — stash the elided fields (or just downgraded_from: "1.7.0") under unmapped, which older schemas tolerate and SIEMs ignore — so the record stays honest about what it dropped? The stripping itself is unavoidable for older consumers; the question is only whether the emitted event should admit it happened, or whether the JSONL is explicitly a lossy export with fidelity kept elsewhere. Either way it's worth a line in the docs so an operator on v1.1 knows AI attribution won't be in that stream.

I'm a bit ambivalent on this. Curious what @zanetworker thinks as he's closest to the use case. I usually default to "doing less" when able. Operators working with OCSF should know what version they are on and also understand there are other versions out there that they are missing the benefits from. IMHO we could call this out in the actual docs pages "OpenShell supports up to version 1.8 in order support newer AI properties, if you need this data you should upgrade." or something to that effect.

Fair, and "do less" is the right default. The one thing a docs upgrade-nudge doesn't cover is the persona split: the nudge helps the operator choosing a version, but the concern is the auditor who later reads a v1.1 line and can't tell "no model" from "model stripped" — they never saw the config, so the doc never reaches them. If the JSONL is meant as a lossy export (real record kept upstream), a docs line nails it and there's nothing more to do. If the v1.1 stream is the retained audit record for a v1.1 shop, that's where a single downgraded_from marker earns its keep — one field, no payload.

[johntmyers]

gator-agent

Blocker Follow-Up Nudge

This PR is still blocked after more than 48 business hours because the current-head Branch E2E Checks workflow has not been rerun with test:e2e active.

Next action: @NVIDIA/openshell-maintainers, open workflow run 32232810685 and choose Re-run all jobs so the required E2E gate can execute for head 12d577fe83d0b7783aa990b3681263c15d781acb.

[github-actions]

Label test:e2e applied for 12d577f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

Author Follow-Up Nudge

This PR has unresolved maintainer feedback about the downgrade documentation, and the pending author action has waited more than 48 business hours. The current page says downgraded JSONL does not lose security-relevant data and remains the complete record, while the downgrade strips ai_model attribution.

@zanetworker, please clarify whether downgraded JSONL is intentionally a lossy export with a full-fidelity record retained elsewhere, or update the documentation/implementation so auditors can distinguish “no model involved” from “model attribution stripped.”