Uh oh!
There was an error while loading. Please reload this page.
fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies - #4532
Conversation
…e sampling applies The internal APM tracer's ParentBasedSampler left remoteParentSampled at its AlwaysOn default, so any request arriving with a sampled traceparent (SDK task-run traces propagated in from running tasks) was recorded in full, ignoring INTERNAL_OTEL_TRACE_SAMPLING_RATE. On api that is ~99.6% of spans, so the divisor was effectively inert. Register a non-inheriting propagator: inject still delegates to W3C trace+baggage so outbound propagation is unchanged, but extract drops the parent span, so every inbound request roots its own trace and the ratio sampler applies uniformly. This also stops api stitching onto (and inflating) the SDK's task-run traces, which is where the untrimmable multi-thousand-span chains came from. Also set remoteParentSampled to the ratio sampler as a fallback.
|
WalkthroughThe webapp tracer now uses typed composite W3C trace-context and baggage propagators. Its non-inheriting propagator removes extracted span contexts while preserving propagation behavior. Provider registration installs this propagator with the existing context manager. Sampling now reuses one ratio sampler for custom root sampling and remote sampled-parent handling. A change note documents reduced internal API overhead under high load. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
What
The internal tracing
ParentBasedSamplerintracer.server.tsleftremoteParentSampledat its default ofAlwaysOn. Any request arriving with atraceparentwhose sampled flag was set got recorded in full, bypassingINTERNAL_OTEL_TRACE_SAMPLING_RATEentirely. Because the SDK propagates its (always-sampled) trace context on calls back to the platform from inside running tasks, the large majority of API server spans inherited a sampled parent and ignored the divisor. The sampling knob was effectively inert on the busiest service.This registers a custom propagator (
NonInheritingTraceContextPropagator) that stops adopting the inbound trace as the parent:injectstill delegates to the standard W3C trace + baggage propagators, so outbound propagation is unchanged.extractdrops the parent span (trace.deleteSpan) while preserving baggage, so every incoming request roots its own trace and the ratio sampler applies uniformly.remoteParentSampledis also set to the ratio sampler as a belt-and-suspenders fallback, in case an inbound sampled parent ever reaches the sampler another way.Two effects: the divisor becomes effective on the API server, and the API no longer stitches onto (and inflates) the propagated task-run traces, which is where the very large, un-thinnable trace chains came from. Rooting each request removes those chains rather than only diluting them.
Only the internal APM trace pipeline (
INTERNAL_OTEL_TRACE_EXPORTER_URL) is affected. The user-facing run-trace pipeline (otel.v1.traces-> ClickHouse) is a separate path and is untouched. The only consumer of the global propagator'sextractis the OTel HTTP/Express auto-instrumentation, so the blast radius is inbound-request trace shape.Evidence (local full-stack red/green, divisor 10)
A local OTLP/JSON sink counting spans; a driver fires N requests at a real endpoint, each carrying a distinct sampled
traceparent, then counts how many spans/traces carry that run's marker.Before: 100% of inherited-sampled requests kept, divisor ignored. After: ~10% kept (the divisor), converging on it at larger N. In every after-run each kept request is a single self-rooted trace (kept spans == kept distinct traces), confirming the inherited chains are gone, not just thinned.
typecheckpasses.Rollout / rollback
No flag. Behavior stays governed by the existing
INTERNAL_OTEL_TRACE_SAMPLING_RATE. Rollback is a straight revert with no data migration.Notes
Internal dashboards that count raw span or request volume from this pipeline will read lower once this ships. That is expected: those counts were inflated by the bypass, not a real drop in traffic. Latency/percentile monitors retain plenty of samples at the current divisor.
refs TRI-13031