feat(vm): export driver traces over OTLP by krishicks · Pull Request #2564 · NVIDIA/OpenShell

krishicks · 2026-07-30T21:13:06Z

Summary

Continue distributed traces across the gateway-to-driver process boundary and export VM driver spans to the same OTLP/gRPC collector. The driver reports as the distinct openshell-driver-vm service.

Updated the gateway architecture and configuration reference with a generic external-driver forwarding contract.

Related Issue

Refs #2507

Changes

Instrumented:

Every RemoteComputeDriver RPC injects the active W3C trace context into tonic metadata. Managed VM readiness and runtime initialization give startup capability probes stable parent operations rather than isolated root spans.
A tonic service layer creates fixed, low-cardinality server spans for every ComputeDriver RPC. New handlers inherit tracing automatically; failures record OpenTelemetry error status and the gRPC status code.
Background provisioning remains attached to CreateSandbox after the RPC returns without extending the RPC span lifetime.
Provisioning records image preparation, bootstrap image resolution, overlay preparation, lifecycle configuration, pre-launch hooks, guest preparation, and launcher spawn as child spans.
VM startup reconciliation roots one trace for the persisted-sandbox scan, with per-sandbox restore and provision operations beneath it. The root remains open until all spawned restore tasks finish.
Delete cleanup records its own child operation.

Design notes:

The gateway forwards its configured OTLP endpoint to managed external drivers. SDK OTEL_* variables continue to own sampling, batching, limits, headers, and transport tuning.
The VM driver has its own tracer provider and service resource so trace backends preserve the service boundary.
RPC operation names come from an explicit method mapping, keeping cardinality bounded without parsing the protobuf descriptor set at runtime.
Propagation uses a remote SpanContext for spawned provisioning. This keeps one trace while allowing the CreateSandbox server span to finish when the RPC response is sent.
Startup restoration is independent of gateway requests. It begins at the VM driver reconciliation span rather than attaching to an unrelated RPC.
Existing tracing events remain on the logging path. The OpenTelemetry layer exports spans only and excludes the SDK exporter callsites to avoid recursive traces.
Export configuration failures do not prevent the driver from serving, and buffered spans are drained during graceful shutdown.
Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC outcomes without recording credentials, sandbox tokens, or request query parameters.

Testing

mise run pre-commit passes
Unit tests added/updated
E2E tests added/updated (if applicable)

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable)

github-actions · 2026-07-30T21:16:27Z

🌿 Preview your docs:https://nvidia-preview-pr-2564.docs.buildwithfern.com/openshell

copy-pr-bot · 2026-07-30T22:09:54Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

copy-pr-bot · 2026-07-30T22:57:44Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

matthewgrossman

Generated with help from Codex.

matthewgrossman

Generated with help from Codex.

Continue distributed traces across the gateway-to-driver process boundary and export VM driver spans to the same OTLP/gRPC collector. The driver reports as the distinct openshell-driver-vm service. Updated the gateway architecture and configuration reference with a generic external-driver forwarding contract. Instrumented: - Every RemoteComputeDriver RPC injects the active W3C trace context into tonic metadata. Managed VM readiness and runtime initialization give startup capability probes stable parent operations rather than isolated root spans. - A tonic service layer creates fixed, low-cardinality server spans for every ComputeDriver RPC. New handlers inherit tracing automatically; failures record OpenTelemetry error status and the gRPC status code. - Background provisioning remains attached to CreateSandbox after the RPC returns without extending the RPC span lifetime. - Provisioning records image preparation, bootstrap image resolution, overlay preparation, lifecycle configuration, pre-launch hooks, guest preparation, and launcher spawn as child spans. - VM startup reconciliation roots one trace for the persisted-sandbox scan, with per-sandbox restore and provision operations beneath it. The root remains open until all spawned restore tasks finish. - Delete cleanup records its own child operation. Design notes: - The gateway forwards its configured OTLP endpoint to managed external drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits, headers, and transport tuning. - The VM driver has its own tracer provider and service resource so trace backends preserve the service boundary. - RPC operation names come from an explicit method mapping, keeping cardinality bounded without parsing the protobuf descriptor set at runtime. - Propagation uses a remote SpanContext for spawned provisioning. This keeps one trace while allowing the CreateSandbox server span to finish when the RPC response is sent. - Startup restoration is independent of gateway requests. It begins at the VM driver reconciliation span rather than attaching to an unrelated RPC. - Existing tracing events remain on the logging path. The OpenTelemetry layer exports spans only and excludes the SDK exporter callsites to avoid recursive traces. - Export configuration failures do not prevent the driver from serving, and buffered spans are drained during graceful shutdown. - Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC outcomes without recording credentials, sandbox tokens, or request query parameters. Refs #2507 Signed-off-by: Kris Hicks <khicks@nvidia.com>

krishicks · 2026-08-03T19:20:05Z

I think all the feedback has been addressed.

matthewgrossman

lgtm1

krishicks requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 30, 2026 21:13

krishicks changed the title ~~feat(vm): export driver traces over OTLP~~feat(vm): export vm driver traces over OTLPJul 30, 2026

krishicks changed the title ~~feat(vm): export vm driver traces over OTLP~~feat(vm): export driver traces over OTLPJul 30, 2026

krishicks force-pushed the hicks/push-nkmtsusunlnk branch from 498b495 to 32b368bCompare July 30, 2026 21:39

krishicks marked this pull request as draft July 30, 2026 22:09

krishicks force-pushed the hicks/push-nkmtsusunlnk branch from 32b368b to 47cd4ebCompare July 30, 2026 22:57

krishicks force-pushed the hicks/push-nkmtsusunlnk branch 2 times, most recently from 72dfb6a to 43cce78Compare July 31, 2026 20:05

krishicks marked this pull request as ready for review July 31, 2026 20:06

krishicks force-pushed the hicks/push-nkmtsusunlnk branch 4 times, most recently from 5faf418 to c3de0c8Compare August 3, 2026 15:50

SDAChess reviewed Aug 3, 2026
View reviewed changes

Comment threadbazel-bin Outdated

krishicks force-pushed the hicks/push-nkmtsusunlnk branch 3 times, most recently from 9b888d6 to d4ae496Compare August 3, 2026 16:46

SDAChess reviewed Aug 3, 2026
View reviewed changes

Comment threadcrates/openshell-server/src/compute/vm.rs

krishicks force-pushed the hicks/push-nkmtsusunlnk branch from d4ae496 to 2bad42fCompare August 3, 2026 16:54

matthewgrossman reviewed Aug 3, 2026
View reviewed changes

Comment threadcrates/openshell-driver-vm/src/driver.rs Outdated
Comment threadcrates/openshell-driver-vm/src/main.rs

matthewgrossman reviewed Aug 3, 2026
View reviewed changes

Comment threadcrates/openshell-driver-vm/src/otel_tracing.rs
Comment threadcrates/openshell-driver-vm/src/driver.rs

krishicks force-pushed the hicks/push-nkmtsusunlnk branch from 2bad42f to 98659dbCompare August 3, 2026 18:09

krishicks force-pushed the hicks/push-nkmtsusunlnk branch from 98659db to d50ac12Compare August 3, 2026 18:40

matthewgrossman approved these changes Aug 3, 2026
View reviewed changes

krishicks enabled auto-merge August 3, 2026 20:52

krishicks added this pull request to the merge queue Aug 3, 2026

Merged via the queue into main with commit 8328412Aug 3, 2026
31 checks passed

krishicks deleted the hicks/push-nkmtsusunlnk branch August 3, 2026 21:03

Navigation Menu

Uh oh!

feat(vm): export driver traces over OTLP - #2564

feat(vm): export driver traces over OTLP#2564
krishicks merged 1 commit into
mainfrom
hicks/push-nkmtsusunlnk

krishicks commented Jul 30, 2026•
edited
Loading

Uh oh!

github-actionsBot commented Jul 30, 2026

Uh oh!

copy-pr-botBot commented Jul 30, 2026

Uh oh!

copy-pr-botBot commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

matthewgrossman left a comment

Uh oh!

Uh oh!

Uh oh!

matthewgrossman left a comment

Uh oh!

Uh oh!

Uh oh!

krishicks commented Aug 3, 2026

Uh oh!

matthewgrossman left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

krishicks commented Jul 30, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

github-actionsBot commented Jul 30, 2026

Uh oh!

copy-pr-botBot commented Jul 30, 2026

Uh oh!

copy-pr-botBot commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

matthewgrossman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

matthewgrossman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

krishicks commented Aug 3, 2026

Uh oh!

matthewgrossman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

krishicks commented Jul 30, 2026•
edited
Loading