Skip to content

Add OpenTelemetry trace context propagation - #99

Merged
patrickleet merged 4 commits into
codex/prometheus-metrics-endpointfrom
codex/opentelemetry-tracing-compatibility
Jul 5, 2026
Merged

patrickleet merged 4 commits into
codex/prometheus-metrics-endpointfrom
codex/opentelemetry-tracing-compatibility

Conversation

@patrickleet

@patrickleet patrickleet commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add canonical W3C trace-context metadata helpers and carrier APIs for Message, Entity, EventRecord, and OutboxMessage
  • preserve traceparent/tracestate through Knative ingress, outbox-to-message mapping, direct dispatch, durable event storage, and broker metadata round-trips
  • add an optional distributed otel feature that emits framework-owned tracing spans without requiring an OpenTelemetry SDK in the default build
  • extend ServiceManifest plus dsvc scaffold --metrics --tracing/--otel GitOps output for OTLP env values and HTTP ServiceMonitor generation
  • document the Distributed/Hops boundary for OTLP export to ObserveStack Alloy/Tempo

Verification

  • cargo fmt --check
  • git diff --check
  • cargo test --all-features
  • cargo clippy --lib --all-features -- -D warnings
  • cargo test -p distributed_cli
  • cargo clippy -p distributed_cli --all-targets -- -D warnings
  • cargo run -p distributed_cli --bin dsvc -- scaffold observability-orders --path target/tmp/scaffold-observability --store in-memory --transport http --metrics --tracing --gitops --force --distributed-path .
  • cargo check --manifest-path target/tmp/scaffold-observability/Cargo.toml

Implements [[tasks/opentelemetry-tracing-compatibility]].

Summary by CodeRabbit

  • New Features
    • Added optional service observability to dctl scaffold: tracing/otel (OTLP-enabled spans with GitOps/Helm env injection) and metrics (Prometheus /metrics artifacts for HTTP).
    • Introduced W3C trace-context (traceparent/tracestate) propagation across messages, entities, event records, and outbox flows.
  • Bug Fixes
    • Improved trace-context preservation and precedence for transport round-trips and CloudEvents (HTTP header values preferred when present).
  • Documentation
    • Added observability guidance detailing trace-context behavior and otel span configuration.
  • Tests
    • Expanded end-to-end compile and transport/integration tests to validate tracing and trace-context round trips.

@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a0c29ecb-7c84-4d07-9ae0-93ffada06573

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR adds trace-context propagation helpers, observability manifest types, runtime tracing spans, and scaffold support for metrics and tracing. It also updates generated GitOps output, docs, and transport tests to carry and validate trace metadata.

Changes

Observability and scaffolded tracing

Layer / File(s) Summary
Trace context core
Cargo.toml, src/trace_context.rs, src/lib.rs
Adds the optional otel feature, defines trace-context helpers and validation, and re-exports trace-context types and constants at the crate root.
Service observability manifest
src/manifest.rs, src/lib.rs
Adds service observability fields, metrics and tracing manifest types, trace propagation/export enums, and serialization tests.
Metadata carriers
src/bus/message.rs, src/entity/entity.rs, src/entity/event_record.rs, src/outbox/message.rs, docs/transports.md, docs/observability.md
Adds trace-context injection and extraction across message carriers, switches correlation and causation keys to shared constants, and documents metadata propagation.
Dispatch and publish spans
src/bus/runner.rs, src/microsvc/knative_ingress.rs, src/microsvc/service.rs, src/outbox_worker/outbox_dispatch.rs, docs/observability.md
Adds tracing spans around transport receive, CloudEvent parsing, microsvc dispatch, handler invocation, and outbox publish paths, with supporting runtime tests.
Scaffold inputs and spec wiring
distributed_cli/src/lib.rs, distributed_cli/src/cli.rs, distributed_cli/src/generate/mod.rs, distributed_cli/README.md
Adds scaffold flags, extends the public scaffold spec, threads tracing through generator state, and updates scaffold documentation.
Generated service and GitOps output
distributed_cli/src/generate/service_crate.rs, distributed_cli/src/generate/gitops.rs, distributed_cli/src/generate/mod.rs, distributed_cli/tests/cli_scaffold_compile.rs, docs/observability.md
Generates tracing-aware service templates, GitOps values and workload manifests, and compile coverage for traced scaffold output.
Transport and persistence tests
tests/kafka_transport/main.rs, tests/nats_transport/main.rs, tests/persistent_repository_conformance/scenario.rs, tests/rabbitmq_transport/main.rs
Verifies traceparent propagation across Kafka, NATS, RabbitMQ, and repository persistence round-trips.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding OpenTelemetry trace-context propagation support.
Docstring Coverage ✅ Passed Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/opentelemetry-tracing-compatibility

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (2)
distributed_cli/src/generate/service_crate.rs (1)

82-132: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: collapse the duplicated metrics main branches.

The Http if self.metrics and Knative if self.metrics arms are identical except for router vs cloud_events_router. The body (listener bind, /metrics route, axum::serve, metrics() handler) could be generated once with the router constructor selected by transport.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@distributed_cli/src/generate/service_crate.rs` around lines 82 - 132, The
metrics-enabled HTTP and Knative branches in service_crate::generate are
duplicating the same main/metrics handler logic. Refactor the
ServiceTransport::Http if self.metrics and ServiceTransport::Knative if
self.metrics arms to share one template, and make only the router constructor
differ between distributed::microsvc::router and
distributed::microsvc::cloud_events_router while keeping the listener bind,
/metrics route, axum::serve call, and metrics() output in one generated body.
distributed_cli/src/cli.rs (1)

74-74: 📐 Maintainability & Code Quality | 🔵 Trivial

Use visible_alias for the --otel flag to ensure it appears in --help.

The alias = "otel" attribute creates a functional alias that works at runtime but hides it from the help output. Since the README documents --otel as a supported flag, users following the documentation and then running --help will not find it listed.

-    #[arg(long, alias = "otel")]
+    #[arg(long, visible_alias = "otel")]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@distributed_cli/src/cli.rs` at line 74, The `--otel` flag alias on the CLI
argument is currently hidden from help output because `alias = "otel"` is used
in the `cli.rs` argument definition. Update the `#[arg(...)]` on the affected
option in `distributed_cli::cli::Cli` to use `visible_alias = "otel"` instead,
so the flag remains functional and is shown in `--help`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@distributed_cli/src/generate/gitops.rs`:
- Around line 119-131: The OTEL_SERVICE_NAME env var in the gitops template is
using the deploy chart name from .Chart.Name, which produces the wrong service
identity for traces. Update the env block in the generator that builds the Helm
snippet to inject the package/service name already used elsewhere in this
codepath (matching ServiceManifest::new and the service label), instead of
hardcoding .Chart.Name, so trace and metric identities stay aligned.

In `@distributed_cli/src/generate/service_crate.rs`:
- Around line 53-55: The generated main entrypoint is missing tracing setup, so
spans are never exported even when the otel feature is enabled. Update
service_crate.rs in main_rs to conditionally emit the opentelemetry OTLP
exporter initialization from OTEL_EXPORTER_OTLP_ENDPOINT and install the
matching tracing_subscriber layer when self.tracing is true. Make sure the
generated main function performs this setup before calling
distributed::microsvc::serve, and keep the logic gated so non-tracing builds
remain unchanged.

In `@src/microsvc/knative_ingress.rs`:
- Around line 96-113: The trace context update in
parse_cloud_event/inject_http_trace_context is unconditional and can wipe
existing metadata from parse_binary or parse_structured when HTTP trace headers
are absent. Add a guard in inject_http_trace_context so it only calls
TraceContext::inject_vec when at least one of TRACEPARENT or TRACESTATE is
present in the HeaderMap, otherwise leave the Message.metadata unchanged.

In `@src/trace_context.rs`:
- Around line 101-109: The is_valid_traceparent validator currently accepts any
non-ff version and uppercase hex, but it should enforce the documented
version-00 shape. Update the checks in is_valid_traceparent (and the related
logic around the traceparent parsing/validation path) to require version == "00"
and to only accept lowercase hex for version-00 fields, while keeping the
existing length and format validation for trace_id, parent_id, and flags.

---

Nitpick comments:
In `@distributed_cli/src/cli.rs`:
- Line 74: The `--otel` flag alias on the CLI argument is currently hidden from
help output because `alias = "otel"` is used in the `cli.rs` argument
definition. Update the `#[arg(...)]` on the affected option in
`distributed_cli::cli::Cli` to use `visible_alias = "otel"` instead, so the flag
remains functional and is shown in `--help`.

In `@distributed_cli/src/generate/service_crate.rs`:
- Around line 82-132: The metrics-enabled HTTP and Knative branches in
service_crate::generate are duplicating the same main/metrics handler logic.
Refactor the ServiceTransport::Http if self.metrics and
ServiceTransport::Knative if self.metrics arms to share one template, and make
only the router constructor differ between distributed::microsvc::router and
distributed::microsvc::cloud_events_router while keeping the listener bind,
/metrics route, axum::serve call, and metrics() output in one generated body.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: aacf9019-2f39-41e1-8653-f91da7554bfa

📥 Commits

Reviewing files that changed from the base of the PR and between 2e83175 and b07d1eb.

📒 Files selected for processing (24)
  • Cargo.toml
  • distributed_cli/README.md
  • distributed_cli/src/cli.rs
  • distributed_cli/src/generate/gitops.rs
  • distributed_cli/src/generate/mod.rs
  • distributed_cli/src/generate/service_crate.rs
  • distributed_cli/src/lib.rs
  • docs/observability.md
  • docs/transports.md
  • src/bus/message.rs
  • src/bus/runner.rs
  • src/entity/entity.rs
  • src/entity/event_record.rs
  • src/lib.rs
  • src/manifest.rs
  • src/microsvc/knative_ingress.rs
  • src/microsvc/service.rs
  • src/outbox/message.rs
  • src/outbox_worker/outbox_dispatch.rs
  • src/trace_context.rs
  • tests/kafka_transport/main.rs
  • tests/nats_transport/main.rs
  • tests/persistent_repository_conformance/scenario.rs
  • tests/rabbitmq_transport/main.rs

Comment thread distributed_cli/src/generate/gitops.rs Outdated
Comment thread distributed_cli/src/generate/service_crate.rs
Comment thread src/microsvc/knative_ingress.rs
Comment thread src/trace_context.rs Outdated
@patrickleet
patrickleet force-pushed the codex/opentelemetry-tracing-compatibility branch from b07d1eb to 6d43624 Compare June 25, 2026 02:40

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@distributed_cli/README.md`:
- Around line 25-30: Remove the duplicated `--gitops` entry from the options
list in the README help text so the `dctl scaffold` flags are listed only once.
Update the sentence that enumerates common flags around `--gitops`, `--metrics`,
and `--tracing` to keep the list accurate and avoid implying multiple distinct
`--gitops` switches.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ad3470a8-d2c1-499f-8dc1-f77e2428607e

📥 Commits

Reviewing files that changed from the base of the PR and between b07d1eb and 6d43624.

📒 Files selected for processing (24)
  • Cargo.toml
  • distributed_cli/README.md
  • distributed_cli/src/cli.rs
  • distributed_cli/src/generate/gitops.rs
  • distributed_cli/src/generate/mod.rs
  • distributed_cli/src/generate/service_crate.rs
  • distributed_cli/src/lib.rs
  • docs/observability.md
  • docs/transports.md
  • src/bus/message.rs
  • src/bus/runner.rs
  • src/entity/entity.rs
  • src/entity/event_record.rs
  • src/lib.rs
  • src/manifest.rs
  • src/microsvc/knative_ingress.rs
  • src/microsvc/service.rs
  • src/outbox/message.rs
  • src/outbox_worker/outbox_dispatch.rs
  • src/trace_context.rs
  • tests/kafka_transport/main.rs
  • tests/nats_transport/main.rs
  • tests/persistent_repository_conformance/scenario.rs
  • tests/rabbitmq_transport/main.rs
✅ Files skipped from review due to trivial changes (3)
  • docs/observability.md
  • docs/transports.md
  • Cargo.toml
🚧 Files skipped from review as they are similar to previous changes (20)
  • tests/kafka_transport/main.rs
  • distributed_cli/src/cli.rs
  • src/outbox_worker/outbox_dispatch.rs
  • tests/persistent_repository_conformance/scenario.rs
  • src/lib.rs
  • distributed_cli/src/lib.rs
  • tests/rabbitmq_transport/main.rs
  • src/outbox/message.rs
  • distributed_cli/src/generate/gitops.rs
  • src/entity/entity.rs
  • src/microsvc/service.rs
  • tests/nats_transport/main.rs
  • src/microsvc/knative_ingress.rs
  • src/trace_context.rs
  • src/bus/runner.rs
  • src/bus/message.rs
  • distributed_cli/src/generate/service_crate.rs
  • distributed_cli/src/generate/mod.rs
  • src/entity/event_record.rs
  • src/manifest.rs

Comment thread distributed_cli/README.md Outdated
@patrickleet
patrickleet force-pushed the codex/opentelemetry-tracing-compatibility branch 2 times, most recently from 711ce33 to 258708c Compare June 25, 2026 03:01
@patrickleet

Copy link
Copy Markdown
Collaborator Author

CodeRabbit summary nitpick follow-up (--otel alias): fixed in 258708c. --tracing now uses visible_alias = "otel", and scaffold_help_lists_otel_alias asserts generated help shows both flags.

@patrickleet

Copy link
Copy Markdown
Collaborator Author

CodeRabbit summary nitpick follow-up (duplicated metrics main branches): fixed in 258708c. Generated metrics entrypoints now share one template and select the appropriate router by transport.

@patrickleet
patrickleet force-pushed the codex/opentelemetry-tracing-compatibility branch from 258708c to 7782aa9 Compare July 4, 2026 22:29
@patrickleet
patrickleet changed the base branch from main to codex/prometheus-metrics-endpoint July 4, 2026 22:29
@patrickleet

Copy link
Copy Markdown
Collaborator Author

Rebased and stacked on #100 (base retargeted to codex/prometheus-metrics-endpoint; merges cleanly into main once #100 lands). The two PRs were authored in parallel off the same base and both implemented --metrics scaffolding — #100's implementation is kept as canonical:

Verification: fmt, clippy (lib all-features + cli all-targets) -D warnings, cargo test --all-features (783 passed), cargo test -p distributed_cli, and scaffold e2e: dctl scaffold observability-orders --store in-memory --transport http --metrics prometheus --tracing --gitops → generated crate compiles with ["http", "metrics", "otel"].

@patrickleet
patrickleet force-pushed the codex/opentelemetry-tracing-compatibility branch from 7782aa9 to 6ac1524 Compare July 4, 2026 22:32
@patrickleet

Copy link
Copy Markdown
Collaborator Author

🤖 Agent dispatch: review-fix

Branch: codex/opentelemetry-tracing-compatibility
Worktree: /private/tmp/worktrees/codex--opentelemetry-tracing-compatibility/distributed
Session: codex--opentelemetry-tracing-compatibility@review-fix@1783207849587-be2c

# Watch live:
atc watch --id codex--opentelemetry-tracing-compatibility@review-fix@1783207849587-be2c
# View logs:
atc logs codex--opentelemetry-tracing-compatibility@review-fix@1783207849587-be2c
# Attach to tmux:
tmux attach -t codex--opentelemetry-tracing-compatibility@review-fix@1783207849587-be2c

@patrickleet

Copy link
Copy Markdown
Collaborator Author

Review-fix pass completed and pushed.

Commit Comment Fix
44ec4e9 Major: generated --tracing scaffolds enabled otel but did not install a tracer/exporter (@coderabbitai) Added generated OTLP/tracing dependencies, subscriber/exporter initialization, server-result handling, tracer provider shutdown, docs, and ignored compile tests for HTTP and Knative tracing scaffolds.
44ec4e9 Nit: generated main.rs had duplicated transport branches (@coderabbitai) Refactored the scaffold generator to share the main template and vary only the serve block.
44ec4e9 Minor: CloudEvents trace header injection duplicated trace carrier replacement logic (@coderabbitai) Reused TraceContext::inject_vec and kept the guard that preserves structured CloudEvent trace extensions when HTTP trace headers are absent.
44ec4e9 Follow-up review: tracing Helm env condition used left-trim markers inside an env: list Removed the trim markers so rendered YAML cannot lose the preceding newline; added a generator assertion for this shape.
44ec4e9 Clippy: metrics test lock was held across an .await Converted that test to a synchronous test with a local block_on helper so the global metrics test lock still serializes state without crossing an await point.

Build status:

  • cargo fmt --check passed
  • git diff --check passed
  • cargo test --workspace --all-targets passed
  • cargo test --workspace --all-features --all-targets --verbose passed
  • cargo test -p distributed_cli --verbose -- --include-ignored passed
  • cargo clippy --workspace --all-targets --all-features -- -D warnings passed
  • cargo hack check --workspace --each-feature --verbose passed

Prefetched review data showed 0 unresolved threads before this push, so no thread resolution calls were needed.

Confidence: 100%. No human-review blockers flagged.

@coderabbitai

coderabbitai Bot commented Jul 4, 2026

Copy link
Copy Markdown

@patrickleet Thanks for the detailed write-up and the thorough verification pass. Nice catch handling the tracer/exporter installation for --tracing scaffolds — that was the key gap. The rest of the cleanups (shared main.rs template, reused TraceContext::inject_vec, fixed Helm trim markers, and the sync test fix for the clippy lint) all sound solid.

I'll take a look at the updated diff.

ᕦ(ò_ó)ᕤ

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

patrickleet and others added 3 commits July 4, 2026 18:57
Implements [[tasks/opentelemetry-tracing-compatibility]]
Implements [[codex/opentelemetry-tracing-compatibility]]
When a dispatch or outbox publish runs inside an already-active local span,
applying the remote traceparent unconditionally re-parents the framework span
and breaks the local hierarchy. Only extract the remote parent at trace entry
(no current span). Found by codex review of #112; lands here because the
affected code is this PR's.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LRjoCyotvAaK3VbhZRd2gq
@patrickleet
patrickleet force-pushed the codex/opentelemetry-tracing-compatibility branch from 44ec4e9 to df34e5e Compare July 5, 2026 00:02
patrickleet added a commit that referenced this pull request Jul 5, 2026
The library fix (set_span_parent_from_metadata_if_no_current_span) moved to
#99 where the affected code lives; this keeps the regression coverage found
by the codex review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/outbox/message.rs (1)

493-521: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

traceparent()/tracestate() are case-sensitive while trace_context() is not.

meta() does an exact-case HashMap::get, but trace_context() reconstructs via TraceContext::from_metadata, which matches keys case-insensitively (eq_ignore_ascii_case). If metadata ever contains a non-canonically-cased traceparent/tracestate entry (e.g. inserted via set_meta or via a raw HashMap passed to create_with_metadata/encode_with_metadata rather than through set_trace_context), traceparent()/tracestate() silently return None while trace_context() still returns a populated value — the two accessors disagree on the same underlying data. Given the PR's own Message carrier deliberately makes trace lookups case-insensitive to tolerate non-canonical header casing from external systems, this HashMap-backed carrier should apply the same guarantee.

♻️ Proposed fix: make the single-key getters case-insensitive too
-    /// Get the W3C `traceparent`, if set.
-    pub fn traceparent(&self) -> Option<&str> {
-        self.meta(TRACEPARENT)
-    }
-
-    /// Get the W3C `tracestate`, if set.
-    pub fn tracestate(&self) -> Option<&str> {
-        self.meta(TRACESTATE)
-    }
+    /// Get the W3C `traceparent`, if set.
+    pub fn traceparent(&self) -> Option<&str> {
+        self.metadata
+            .iter()
+            .find(|(k, _)| k.eq_ignore_ascii_case(TRACEPARENT))
+            .map(|(_, v)| v.as_str())
+    }
+
+    /// Get the W3C `tracestate`, if set.
+    pub fn tracestate(&self) -> Option<&str> {
+        self.metadata
+            .iter()
+            .find(|(k, _)| k.eq_ignore_ascii_case(TRACESTATE))
+            .map(|(_, v)| v.as_str())
+    }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/outbox/message.rs` around lines 493 - 521, `Message::traceparent()` and
`Message::tracestate()` currently use `meta()` for exact-case lookup, which can
disagree with `Message::trace_context()`; update the single-key accessors to
perform case-insensitive metadata matching like `TraceContext::from_metadata`
does. Reuse the existing `Message`/`meta` path or add a helper so
`traceparent()`, `tracestate()`, and `trace_context()` all return consistent
results regardless of key casing from `set_meta`, `create_with_metadata`, or
`encode_with_metadata`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/outbox/message.rs`:
- Around line 493-521: `Message::traceparent()` and `Message::tracestate()`
currently use `meta()` for exact-case lookup, which can disagree with
`Message::trace_context()`; update the single-key accessors to perform
case-insensitive metadata matching like `TraceContext::from_metadata` does.
Reuse the existing `Message`/`meta` path or add a helper so `traceparent()`,
`tracestate()`, and `trace_context()` all return consistent results regardless
of key casing from `set_meta`, `create_with_metadata`, or
`encode_with_metadata`.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5c9103e8-9ad1-4d9d-b37a-2786c50cda78

📥 Commits

Reviewing files that changed from the base of the PR and between 6d43624 and 44ec4e9.

📒 Files selected for processing (25)
  • Cargo.toml
  • distributed_cli/README.md
  • distributed_cli/src/cli.rs
  • distributed_cli/src/generate/gitops.rs
  • distributed_cli/src/generate/mod.rs
  • distributed_cli/src/generate/service_crate.rs
  • distributed_cli/src/lib.rs
  • distributed_cli/tests/cli_scaffold_compile.rs
  • docs/observability.md
  • docs/transports.md
  • src/bus/message.rs
  • src/bus/runner.rs
  • src/entity/entity.rs
  • src/entity/event_record.rs
  • src/lib.rs
  • src/manifest.rs
  • src/microsvc/knative_ingress.rs
  • src/microsvc/service.rs
  • src/outbox/message.rs
  • src/outbox_worker/outbox_dispatch.rs
  • src/trace_context.rs
  • tests/kafka_transport/main.rs
  • tests/nats_transport/main.rs
  • tests/persistent_repository_conformance/scenario.rs
  • tests/rabbitmq_transport/main.rs
💤 Files with no reviewable changes (5)
  • tests/persistent_repository_conformance/scenario.rs
  • tests/nats_transport/main.rs
  • tests/rabbitmq_transport/main.rs
  • tests/kafka_transport/main.rs
  • src/trace_context.rs
✅ Files skipped from review due to trivial changes (3)
  • distributed_cli/README.md
  • docs/transports.md
  • docs/observability.md
🚧 Files skipped from review as they are similar to previous changes (9)
  • distributed_cli/src/lib.rs
  • src/bus/runner.rs
  • src/entity/event_record.rs
  • src/entity/entity.rs
  • src/microsvc/service.rs
  • src/microsvc/knative_ingress.rs
  • src/lib.rs
  • src/bus/message.rs
  • src/manifest.rs

* test: docker-level observability integration suites

- tests/metrics_exposition: real HTTP scrape of /metrics across all framework
  families, linted with promtool check metrics (PROMTOOL env gate)
- tests/otel_export: real OTLP pipeline -> OpenTelemetry Collector container;
  asserts distributed.microsvc.dispatch arrives parented to the incoming
  W3C traceparent (endpoint/file env gates)
- integration-observability.yaml: reusable workflow running both suites plus
  helm template + kubeconform validation of scaffolded ServiceMonitor/
  PrometheusRule/OTLP-env output against published CRD schemas; wired into
  PR-quality and push-main pipelines

Implements [[tasks/observability-integration-tests]]

* test: cover nested span parenting in the OTLP export e2e

The library fix (set_span_parent_from_metadata_if_no_current_span) moved to
#99 where the affected code lives; this keeps the regression coverage found
by the codex review.
@patrickleet
patrickleet merged commit a3fba9b into codex/prometheus-metrics-endpoint Jul 5, 2026
12 checks passed
patrickleet added a commit that referenced this pull request Jul 7, 2026
* Add Prometheus metrics endpoint

* fix: address prometheus metrics review feedback

Implements [[codex/prometheus-metrics-endpoint]]

* Add OpenTelemetry trace context propagation (#99)

* feat: add OpenTelemetry trace context propagation

Implements [[tasks/opentelemetry-tracing-compatibility]]

* fix: resolve tracing scaffold review findings

Implements [[codex/opentelemetry-tracing-compatibility]]

* fix: preserve local span hierarchy when a parent span is active

When a dispatch or outbox publish runs inside an already-active local span,
applying the remote traceparent unconditionally re-parents the framework span
and breaks the local hierarchy. Only extract the remote parent at trace entry
(no current span). Found by codex review of #112; lands here because the
affected code is this PR's.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LRjoCyotvAaK3VbhZRd2gq

* test: docker-level observability integration suites (#112)

* test: docker-level observability integration suites

- tests/metrics_exposition: real HTTP scrape of /metrics across all framework
  families, linted with promtool check metrics (PROMTOOL env gate)
- tests/otel_export: real OTLP pipeline -> OpenTelemetry Collector container;
  asserts distributed.microsvc.dispatch arrives parented to the incoming
  W3C traceparent (endpoint/file env gates)
- integration-observability.yaml: reusable workflow running both suites plus
  helm template + kubeconform validation of scaffolded ServiceMonitor/
  PrometheusRule/OTLP-env output against published CRD schemas; wired into
  PR-quality and push-main pipelines

Implements [[tasks/observability-integration-tests]]

* test: cover nested span parenting in the OTLP export e2e

The library fix (set_span_parent_from_metadata_if_no_current_span) moved to
#99 where the affected code lives; this keeps the regression coverage found
by the codex review.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix: address observability code-review findings

- knative ingress: tracestate-only HTTP headers no longer delete the
  message's existing traceparent (headers win only with a traceparent)
- backlog gauges: drop the 5s refresh throttle — refreshes are
  activity-driven with no timer, so any skipped pass froze the gauges
  at stale values after the final drain
- OutboxStore::backlog_stats default: bounded scan (1000 rows) instead
  of paging the whole outbox; count saturates, oldest stays exact
- span parenting: one rule everywhere — an active local span wins;
  transport receive now uses the same conditional parenting as
  dispatch/outbox, documented on the helper
- TraceContext::from_metadata: first match wins on duplicate keys,
  matching Message accessors and OTel span-parent extraction
- tests: serialize unknown_command against the global metrics registry

Implements [[tasks/observability-prs-rebase]]

* fix: restrict scaffolded OTLP protocol

Implements [[codex/prometheus-metrics-endpoint]]

* chore: prepare telemetry foundation

Implements [[tasks/distributed-telemetry-foundation-cleanup-1]]

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant