Uh oh!
There was an error while loading. Please reload this page.
Guard core-metric telemetry drain against queue-full delta loss - #12081
Guard core-metric telemetry drain against queue-full delta loss#12081dougqh wants to merge 2 commits into
Conversation
getValueAndReset() resets a counter's delta baseline, so calling it before the offer() succeeds means a full metricsQueue silently drops that delta forever -- worst under telemetry backpressure when the agent is unreachable. In both the span-metrics and baggage collection loops, check remainingCapacity() before reading each counter and stop early instead; the untouched counters are collected on the next cycle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Fills CoreMetricCollector's drain past its ArrayBlockingQueue capacity and verifies every counter's delta is reported exactly once across cycles. Fails against the pre-fix code (counters read-and-reset while the queue was full were dropped) and passes with the remainingCapacity() guard. Robust to the shared SpanMetricRegistryImpl singleton: tags its own instrumentations with a unique prefix, counts only those, and drains cycle-by-cycle until the queue is empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
More details
The capacity checks preserve both span-metric and baggage deltas when the telemetry queue is full: a compiled harness retained 1,224 span deltas and two baggage deltas across the overflow cycle, with no unexpected output or loss observed. The repository Gradle test could not start because the sandbox lacks the required Java 25 toolchain and cannot download it.
📊 Validated against 2 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit b55cb17 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
What Does This Do?
Stops the core-metric telemetry drain early when its queue is full instead of continuing to read counters it can't enqueue. Concretely: check
remainingCapacity()before reading each counter in the span-metrics and baggage loops, and break before thegetValueAndReset()+CoreMetricallocation when the queue is full. Behavior is unchanged in the healthy (non-full) case.This does two things under overload:
CoreMetricobjects (and their tag strings) only to drop them on a failedoffer().getValueAndReset()resets the counter's baseline, so a value read but not enqueued was lost permanently. Untouched counters now keep their accumulated delta and are collected on the next cycle.Motivation
Both effects matter under backpressure: when the Datadog agent is unreachable, telemetry backs up and the queue fills. The original goal was to cut the per-counter allocation while the system is already overloaded. In doing that it became clear there's also a real data-loss bug — the counts we most want (the ones describing the degraded state) are exactly the ones the old code silently discarded.
Additional Notes
CoreMetricCollector.prepareMetrics()drains each core counter withgetValueAndReset()and thenoffer()s the resultingCoreMetriconto a fixedArrayBlockingQueue. BecausegetValueAndReset()resets the counter's delta baseline, if the queue is already full the reset has happened but the value is dropped — that delta is lost permanently, not deferred to the next cycle.Sole-producer invariant:
prepareMetrics()is the only writer to the queue (drain()only removes), so aremainingCapacity() == 0check followed byoffer()cannot race into a lost delta.Scope / relation to #12070
The identical guard for the client-side trace-stats collapse loop ships in #12070 (which introduced that loop). This PR fixes the two pre-existing loops (span metrics, baggage) that share the same pattern.
🤖 Generated with Claude Code