Uh oh!
There was an error while loading. Please reload this page.
Add span-creation JMH benchmarks (front-half allocation) - #11915
Conversation
Adds application-thread (front-half) allocation benchmarks that ground the TagMap 2.0 / SpanPrototype work against the real span lifecycle: - SpanCreationBenchmark (front-A): single span create -> tag -> finish. Bare baseline (startSpan vs buildSpan), two known-tag shapes set after start — web-server (7 tags) and JDBC/DB client (9 tags) — plus a builder-tag-path arm (withTag before start, the OTel-bridge shape) to track how the startSpan / buildSpan lineages diverge across releases. - TraceAssemblyBenchmark (front-B): web-shape root + N children (childCount 1/5/20), exercising the per-span baseline-tag copy that map-to-map copy (TagMap 1.0) and the trace/span tag split (level-split) target. - DropWriter: shared no-op Writer so finish() excludes serialization / agent I/O, isolating front-half allocation for -prof gc. Deliberately drift-stable (v1.53->master API) so the same file can be grafted onto old release tags for a historical allocation curve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runs startSpan create+finish on a virtual thread — the regime the platform-thread SpanCreationBenchmark is blind to. startSpan's thread-local SpanBuilder reuse (#9537) is disabled on virtual threads, so a builder is allocated per span there; the builder bypass (#9998) removes it. This bench is where that difference shows. Starts the virtual thread via reflection (Thread.startVirtualThread) so the jmh source set still compiles on pre-21 toolchains; requires a 21+ JDK at run time. The per-op vthread spawn/join cost is constant across tracer versions, so it cancels in the version-over-version delta. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Fold the v1.53.0 -> 1.64.0 sweep results into the SpanCreationBenchmark and TraceAssemblyBenchmark javadoc so the trend is discoverable from the source without re-running. Allocation (gc.alloc.rate.norm B/op) carries a net Δ%; throughput (ops/us) is included but flagged directional-only and given no Δ% since it has no reliable trend (laptop thermals + per-fork inlining bimodality). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
More details
Pure benchmark-only PR (no production code changes): all four new JMH files are measurement infrastructure. Every critical API contract was verified statically — DropWriter correctly implements all five Writer methods, all Tags constants exist, asChildOf(root) compiles because CoreTracer.buildSpan() returns the concrete CoreSpanBuilder (which has asChildOf(AgentSpan)) not the SpanBuilder interface (which has only asChildOf(AgentSpanContext)), and the Thread.startVirtualThread MethodHandle pattern was confirmed correct on JDK 21. No behavioral regressions are possible: this PR adds only benchmarks.
📊 Validated against 6 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit e92967b · What is Autotest? · Any feedback? Reach out in #autotest
A real jdbc span is never "servlet.request"; using a distinct "database.query" operation name keeps the web- and DB-shaped span benchmarks from being conflated by operation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Now that the DB-shaped span has its own JDBC_OPERATION_NAME, name the web/server operation constant symmetrically for clarity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sarahchen6
left a comment
There was a problem hiding this comment.
Left a few comments, but otherwise looks good. Pre-approving !
| */ | ||
| final class DropWriter implements Writer { | ||
| @Override | ||
| public void write(List<DDSpan> trace) {} |
There was a problem hiding this comment.
should this writer leverage JMH's blackhole like PendingTraceWrite and BlackholeWriter? IIUC this will keep unused code active for the benchmarks
There was a problem hiding this comment.
Yeah, I think that makes sense. I'd held off initially because I was debating having a way to cover both the foreground / application thread work and the background / serializer & stats thread work.
But either way, using a Blackhole here probably makes sense.
I'm confident that spans weren't escaping, so the results shouldn't change.
So I think I'm going to land the PR as is -- then I'll do a re-run over night just to confirm that the results haven't changed significantly. If they have I'll do another PR to refresh the results.
| * | ||
| * <p>Same conventions as {@link SpanCreationBenchmark}: {@link DropWriter} isolates front-half | ||
| * allocation; read alloc ({@code -prof gc}) as the anchor, throughput as directional; logging must | ||
| * be forced to WARN or DEBUG-line allocation corrupts the numbers. Drift-stable v1.53→master |
There was a problem hiding this comment.
A point brought up by Codex is that all of these benchmarks inherit from https://github.com/DataDog/dd-trace-java/blob/master/dd-trace-core/src/test/resources/logback.xml#L12 which sets the logging level to DEBUG, but at least TraceAssemblyBenchmark here expects logging to be at the WARN level to not corrupt numbers. The recommended fix is to neutralize the logging setting for all of these benchmarks with:
@Fork(value = 3, jvmArgsAppend = "-DTEST_LOG_LEVEL=warn")
WDYT?
There was a problem hiding this comment.
Yeah, that's a good point -- and is a problem that I ended up hitting in another Claude session. Might as well just fix it for good.
| * no serialization or agent I/O leaking into the {@code -prof gc} number. | ||
| * | ||
| * <p>Drift-stable: implements only the five-method {@link Writer} interface, unchanged | ||
| * v1.53→master. |
There was a problem hiding this comment.
"master" is used in descriptions throughout, but this reference will always be changing -- I think we should specify 1.64 instead to be clear that that's what the latest tested version is, or something like "master at the time of this commit"
There was a problem hiding this comment.
The reference to master is largely intentional. My aim is that this benchmark stays valid as a way compare performance over time -- basically as long as we maintain binary compatibility.
Benchmarks that depend on newer APIs will go in their own benchmark classes, since they won't work on older dd-trace-java versions.
… benchmarks DropWriter.write() was a true no-op, letting the JIT treat the finish()-triggered write as dead code; hand the trace to a Blackhole instead. Also force -DTEST_LOG_LEVEL=warn on all three benchmark classes' forks, since logback.xml defaults to DEBUG and the benchmarks' own javadoc requires WARN to keep DEBUG logging from corrupting the -prof gc allocation numbers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/merge |
View all feedbacks in Devflow UI.
The expected merge time in
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
What Does This Do
Adds application-thread ("front-half") allocation benchmarks for the full span-creation lifecycle (create → tag → finish), to ground the TagMap 2.0 / SpanPrototype allocation work against the real span path.
SpanCreationBenchmark— single span. Bare baseline (startSpanvsbuildSpan), two known-tag shapes set after start — web-server (7 tags) and JDBC/DB client (9 tags) — plus a builder-tag-path arm (withTagbeforestart(), the OTel-bridge shape).TraceAssemblyBenchmark— web-shape root + N children (childCount1/5/20), exercising the per-span baseline-tag copy that map-to-map copy (TagMap 1.0) and the trace/span tag split (level-split) target.SpanCreationVirtualThreadBenchmark—startSpanon a virtual thread (JDK 21+ via reflection), the regime whereSpanBuilderreuse is disabled.DropWriter— shared no-opWritersofinish()excludes serialization / agent I/O, isolating front-half allocation for-prof gc.Motivation
gc.alloc.rate.norm(B/op) is deterministic and is the primary signal; throughput is directional. The benchmarks are deliberately drift-stable (only API byte-identical v1.53→master), so the same file can be grafted onto past release tags to reconstruct a historical allocation curve.Additional Notes
-prof gc, and force tracer logging to WARN (unsuppressed DEBUG logging allocates and corrupts the numbers).🤖 Generated with Claude Code
Validation — rigorous 8-thread version sweep (1.53 → 1.64)
These benchmarks were run across every release 1.53→1.64 (
-f3 -wi5 -i5 -t8 -prof gc) to validate the allocation trend. Allocation is the trustworthy axis; throughput is directional (build-to-build inlining-bimodal on a laptop — read trends, not points).Allocation (B/op) — trustworthy axis
bareStartSpanbareBuildSpanwebServerSpanwebServerSpanViaBuilderjdbcClientSpanwebServerTrace[1]webServerTrace[5]webServerTrace[20]Cumulative 1.53 → 1.64:
bareStartSpanbareBuildSpanwebServerSpanwebServerSpanViaBuilderjdbcClientSpanwebServerTrace[1]webServerTrace[5]webServerTrace[20]Two structural steps carry the curve: 1.59 (TagMap 1.0 default → Entry-sharing) and 1.61 (interceptor/links elimination cluster), plus a 1.63 jdbc-specific drop. Single-span −28-31% (triple-confirmed: this sweep, the 4-thread archaeology, and the PetClinic macro stress test at −29%). Trace path smaller (−12-20%) — the per-child fresh-tag insertion cost that TagMap 2.0 (dense + level-split) targets next.
Throughput (ops/us) — directional only
bareStartSpanbareBuildSpanwebServerSpanwebServerSpanViaBuilderjdbcClientSpanwebServerTrace[1]webServerTrace[5]webServerTrace[20]Caveat: throughput on this harness is build-to-build inlining-bimodal (see the bareStart column alternating ~4.2 / ~5.5) — reproducible across clean re-runs, not thermal or contamination, and not stabilized by
@Fork(3)(which averages within a build, not across builds). Read trends, not points. The one real-ish signal is the 1.62→1.64 climb on the tagged arms (web 3.4→4.6, webBuilder 3.5→5.3), corroborating the 1.62isOutbound/span.kind CPU win (#11116) + client-stats changes. Allocation is the axis; throughput corroborates large effects only. A CPU win should be sized from a profile (cycles), not this axis.Measured a real change (#11701)
TraceAssemblyBenchmarkwas used to size #11701 (local children share the parent'sPropagationTags). Isolating just that one-line toggle (share vsempty()-per-child, all else identical), the shared path dropsgc.alloc.rate.normby ~96 B per child (JOL-confirmed shallow size of oneempty()PTags), scaling with fan-out:trace[20]−2031 B/op (≈14.5%),trace[5]−442 B/op, single-span control flat. A concrete consumer validating the per-child slope this harness exists to expose.