Skip to content

Reuse SpanBuilder within a Thread - #9537

Merged
dougqh merged 55 commits into
masterfrom
dougqh/spanbuilder-pooling
Oct 22, 2025
Merged

Reuse SpanBuilder within a Thread#9537
dougqh merged 55 commits into
masterfrom
dougqh/spanbuilder-pooling

Conversation

@dougqh

@dougqhdougqh commented Sep 15, 2025

Copy link
Copy Markdown
Contributor

What Does This Do

This changes reuses a SpanBuilder within a thread.

Motivation

SpanBuilders are one of the major causes of allocation within the client library, but typically only one SpanBuilder is needed at a time in a thread. By reseting and reusing the same SpanBuilder, tracing allocation can be reduced significantly.

By reducing allocation, the garbage collector needs to run less frequently improving the maximum sustainable throughput of the host application. In local tests with Spring Petclinic, this reduces the throughput penalty with tiny heaps (<= 80m) significantly from -40% to -20%. On larger heaps, the gain is more modest, since garbage collector has less of an impact on the application.

In this initial version, the thread local cache isn't used in virtual threads. This restriction exists to avoid creating more memory churn than we save, since virtual threads are created more often than regular threads.

Initial performance experiment
The idea is to store a CoreSpanBuilder per thread, since usually only SpanBuilder is in use at a given time per thread -- and CoreSpanBuilder isn't thread safe
This simple change provides a giant boost in small heaps
Improving Spring petclinic throughput from -39% to -19% with 80m heap
@dougqh
dougqh requested a review from a team as a code ownerSeptember 15, 2025 19:47
@github-actions

github-actionsBot commented Sep 15, 2025

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@dougqhdougqh added comp: core Tracer core type: feature Enhancements and improvements tag: performance Performance related changes labels Sep 15, 2025
@datadog-datadog-prod-us1

datadog-datadog-prod-us1Bot commented Sep 15, 2025

Copy link
Copy Markdown
Contributor

🎯 Code Coverage
Patch Coverage: 42.16%
Total Coverage: 59.68% (-0.04%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3d9c527 | Docs | Was this helpful? Give us feedback!

@pr-commenter

pr-commenterBot commented Sep 15, 2025

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

BaselineCandidate
baseline_or_candidatebaselinecandidate
git_branchmasterdougqh/spanbuilder-pooling
git_commit_date17611562281761157731
git_commit_shacd02b0c3d9c527
release_version1.55.0-SNAPSHOT~cd02b0c3261.54.0-SNAPSHOT~3d9c52713f
See matching parameters
BaselineCandidate
applicationinsecure-bankinsecure-bank
ci_job_date17611594691761159469
ci_job_id11924863111192486311
ci_pipeline_id8000863680008636
cpu_modelIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHzIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_versionLinux runner-zfyrx7zua-project-304-concurrent-0-35u608ts 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/LinuxLinux runner-zfyrx7zua-project-304-concurrent-0-35u608ts 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
moduleAgentAgent
parentNoneNone

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 11 unstable metrics.

Startup time reports for petclinic
gantt
title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.033 s) : 0, 1033376
Total [baseline] (9.815 s) : 0, 9814883
Agent [candidate] (1.027 s) : 0, 1027356
Total [candidate] (10.755 s) : 0, 10754602
section appsec
Agent [baseline] (1.199 s) : 0, 1199341
Total [baseline] (10.962 s) : 0, 10962309
Agent [candidate] (1.201 s) : 0, 1201085
Total [candidate] (10.897 s) : 0, 10897153
section iast
Agent [baseline] (1.166 s) : 0, 1165797
Total [baseline] (11.088 s) : 0, 11087553
Agent [candidate] (1.16 s) : 0, 1159991
Total [candidate] (11.166 s) : 0, 11165763
section profiling
Agent [baseline] (1.177 s) : 0, 1176925
Total [baseline] (10.961 s) : 0, 10960777
Agent [candidate] (1.173 s) : 0, 1172709
Total [candidate] (10.943 s) : 0, 10942802
Loading
  • baseline results
ModuleVariantDurationΔ tracing
Agenttracing1.033 s-
Agentappsec1.199 s165.965 ms (16.1%)
Agentiast1.166 s132.421 ms (12.8%)
Agentprofiling1.177 s143.549 ms (13.9%)
Totaltracing9.815 s-
Totalappsec10.962 s1.147 s (11.7%)
Totaliast11.088 s1.273 s (13.0%)
Totalprofiling10.961 s1.146 s (11.7%)
  • candidate results
ModuleVariantDurationΔ tracing
Agenttracing1.027 s-
Agentappsec1.201 s173.73 ms (16.9%)
Agentiast1.16 s132.635 ms (12.9%)
Agentprofiling1.173 s145.353 ms (14.1%)
Totaltracing10.755 s-
Totalappsec10.897 s142.551 ms (1.3%)
Totaliast11.166 s411.161 ms (3.8%)
Totalprofiling10.943 s188.2 ms (1.7%)
gantt
title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.478 ms) : 0, 1478
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (704.314 ms) : 0, 704314
BytebuddyAgent [candidate] (698.065 ms) : 0, 698065
GlobalTracer [baseline] (246.81 ms) : 0, 246810
GlobalTracer [candidate] (245.778 ms) : 0, 245778
AppSec [baseline] (32.85 ms) : 0, 32850
AppSec [candidate] (32.42 ms) : 0, 32420
Debugger [baseline] (6.422 ms) : 0, 6422
Debugger [candidate] (6.344 ms) : 0, 6344
Remote Config [baseline] (686.034 µs) : 0, 686
Remote Config [candidate] (675.405 µs) : 0, 675
Telemetry [baseline] (9.434 ms) : 0, 9434
Telemetry [candidate] (10.189 ms) : 0, 10189
Flare Poller [baseline] (9.912 ms) : 0, 9912
Flare Poller [candidate] (10.971 ms) : 0, 10971
section appsec
crashtracking [baseline] (1.468 ms) : 0, 1468
crashtracking [candidate] (1.473 ms) : 0, 1473
BytebuddyAgent [baseline] (720.646 ms) : 0, 720646
BytebuddyAgent [candidate] (721.134 ms) : 0, 721134
GlobalTracer [baseline] (235.814 ms) : 0, 235814
GlobalTracer [candidate] (237.164 ms) : 0, 237164
IAST [baseline] (25.103 ms) : 0, 25103
IAST [candidate] (24.938 ms) : 0, 24938
AppSec [baseline] (175.778 ms) : 0, 175778
AppSec [candidate] (175.851 ms) : 0, 175851
Debugger [baseline] (6.106 ms) : 0, 6106
Debugger [candidate] (6.148 ms) : 0, 6148
Remote Config [baseline] (639.31 µs) : 0, 639
Remote Config [candidate] (625.454 µs) : 0, 625
Telemetry [baseline] (8.573 ms) : 0, 8573
Telemetry [candidate] (8.482 ms) : 0, 8482
Flare Poller [baseline] (3.93 ms) : 0, 3930
Flare Poller [candidate] (3.896 ms) : 0, 3896
section iast
crashtracking [baseline] (1.491 ms) : 0, 1491
crashtracking [candidate] (1.467 ms) : 0, 1467
BytebuddyAgent [baseline] (826.036 ms) : 0, 826036
BytebuddyAgent [candidate] (820.823 ms) : 0, 820823
GlobalTracer [baseline] (234.456 ms) : 0, 234456
GlobalTracer [candidate] (234.235 ms) : 0, 234235
IAST [baseline] (27.745 ms) : 0, 27745
IAST [candidate] (26.808 ms) : 0, 26808
AppSec [baseline] (34.442 ms) : 0, 34442
AppSec [candidate] (35.359 ms) : 0, 35359
Debugger [baseline] (6.263 ms) : 0, 6263
Debugger [candidate] (6.195 ms) : 0, 6195
Remote Config [baseline] (618.169 µs) : 0, 618
Remote Config [candidate] (605.222 µs) : 0, 605
Telemetry [baseline] (8.804 ms) : 0, 8804
Telemetry [candidate] (8.752 ms) : 0, 8752
Flare Poller [baseline] (4.233 ms) : 0, 4233
Flare Poller [candidate] (4.194 ms) : 0, 4194
section profiling
crashtracking [baseline] (1.479 ms) : 0, 1479
crashtracking [candidate] (1.484 ms) : 0, 1484
BytebuddyAgent [baseline] (725.92 ms) : 0, 725920
BytebuddyAgent [candidate] (724.954 ms) : 0, 724954
GlobalTracer [baseline] (222.576 ms) : 0, 222576
GlobalTracer [candidate] (222.099 ms) : 0, 222099
AppSec [baseline] (32.767 ms) : 0, 32767
AppSec [candidate] (32.319 ms) : 0, 32319
Debugger [baseline] (8.457 ms) : 0, 8457
Debugger [candidate] (7.448 ms) : 0, 7448
Remote Config [baseline] (742.09 µs) : 0, 742
Remote Config [candidate] (684.549 µs) : 0, 685
Telemetry [baseline] (13.888 ms) : 0, 13888
Telemetry [candidate] (15.121 ms) : 0, 15121
Flare Poller [baseline] (5.011 ms) : 0, 5011
Flare Poller [candidate] (4.193 ms) : 0, 4193
ProfilingAgent [baseline] (111.869 ms) : 0, 111869
ProfilingAgent [candidate] (110.266 ms) : 0, 110266
Profiling [baseline] (112.527 ms) : 0, 112527
Profiling [candidate] (110.886 ms) : 0, 110886
Loading
Startup time reports for insecure-bank
gantt
title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.034 s) : 0, 1033562
Total [baseline] (8.732 s) : 0, 8731760
Agent [candidate] (1.028 s) : 0, 1028126
Total [candidate] (8.743 s) : 0, 8742826
section iast
Agent [baseline] (1.152 s) : 0, 1152100
Total [baseline] (9.363 s) : 0, 9362668
Agent [candidate] (1.159 s) : 0, 1158825
Total [candidate] (9.343 s) : 0, 9342912
Loading
  • baseline results
ModuleVariantDurationΔ tracing
Agenttracing1.034 s-
Agentiast1.152 s118.538 ms (11.5%)
Totaltracing8.732 s-
Totaliast9.363 s630.909 ms (7.2%)
  • candidate results
ModuleVariantDurationΔ tracing
Agenttracing1.028 s-
Agentiast1.159 s130.698 ms (12.7%)
Totaltracing8.743 s-
Totaliast9.343 s600.086 ms (6.9%)
gantt
title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.471 ms) : 0, 1471
crashtracking [candidate] (1.463 ms) : 0, 1463
BytebuddyAgent [baseline] (702.948 ms) : 0, 702948
BytebuddyAgent [candidate] (698.244 ms) : 0, 698244
GlobalTracer [baseline] (247.063 ms) : 0, 247063
GlobalTracer [candidate] (246.268 ms) : 0, 246268
AppSec [baseline] (32.909 ms) : 0, 32909
AppSec [candidate] (32.511 ms) : 0, 32511
Debugger [baseline] (6.447 ms) : 0, 6447
Debugger [candidate] (6.385 ms) : 0, 6385
Remote Config [baseline] (689.943 µs) : 0, 690
Remote Config [candidate] (701.344 µs) : 0, 701
Telemetry [baseline] (9.569 ms) : 0, 9569
Telemetry [candidate] (9.45 ms) : 0, 9450
Flare Poller [baseline] (11.042 ms) : 0, 11042
Flare Poller [candidate] (11.804 ms) : 0, 11804
section iast
crashtracking [baseline] (1.498 ms) : 0, 1498
crashtracking [candidate] (1.487 ms) : 0, 1487
BytebuddyAgent [baseline] (815.611 ms) : 0, 815611
BytebuddyAgent [candidate] (819.544 ms) : 0, 819544
GlobalTracer [baseline] (231.55 ms) : 0, 231550
GlobalTracer [candidate] (233.879 ms) : 0, 233879
IAST [baseline] (26.731 ms) : 0, 26731
IAST [candidate] (26.908 ms) : 0, 26908
AppSec [baseline] (35.357 ms) : 0, 35357
AppSec [candidate] (35.668 ms) : 0, 35668
Debugger [baseline] (6.18 ms) : 0, 6180
Debugger [candidate] (6.164 ms) : 0, 6164
Remote Config [baseline] (601.723 µs) : 0, 602
Remote Config [candidate] (621.737 µs) : 0, 622
Telemetry [baseline] (8.777 ms) : 0, 8777
Telemetry [candidate] (8.852 ms) : 0, 8852
Flare Poller [baseline] (4.299 ms) : 0, 4299
Flare Poller [candidate] (4.194 ms) : 0, 4194
Loading

Load

Parameters

BaselineCandidate
baseline_or_candidatebaselinecandidate
git_branchmasterdougqh/spanbuilder-pooling
git_commit_date17611562281761157731
git_commit_shacd02b0c3d9c527
release_version1.55.0-SNAPSHOT~cd02b0c3261.54.0-SNAPSHOT~3d9c52713f
See matching parameters
BaselineCandidate
applicationinsecure-bankinsecure-bank
ci_job_date17611591671761159167
ci_job_id11924863121192486312
ci_pipeline_id8000863680008636
cpu_modelIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHzIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_versionLinux runner-zfyrx7zua-project-304-concurrent-1-gx4xfptc 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/LinuxLinux runner-zfyrx7zua-project-304-concurrent-1-gx4xfptc 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 4 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.

scenarioΔ mean http_req_durationΔ mean throughputcandidate mean http_req_durationcandidate mean throughputbaseline mean http_req_durationbaseline mean throughput
scenario:load:insecure-bank:profiling:high_loadbetter
[-480.997µs; -180.795µs] or [-5.351%; -2.011%]
unstable
[-47.404op/s; +86.592op/s] or [-9.183%; +16.775%]
8.657ms535.781op/s8.988ms516.188op/s
scenario:load:petclinic:no_agent:high_loadbetter
[-1.538ms; -0.926ms] or [-4.180%; -2.515%]
unstable
[-4.420op/s; +13.095op/s] or [-3.479%; +10.305%]
35.572ms131.412op/s36.804ms127.075op/s
scenario:load:petclinic:code_origins:high_loadbetter
[-2.601ms; -1.753ms] or [-5.722%; -3.856%]
unstable
[-2.240op/s; +12.465op/s] or [-2.175%; +12.106%]
43.277ms108.075op/s45.454ms102.963op/s
scenario:load:petclinic:appsec:high_loadbetter
[-1.995ms; -1.051ms] or [-4.070%; -2.144%]
unstable
[-3.903op/s; +9.928op/s] or [-4.087%; +10.396%]
47.493ms98.513op/s49.015ms95.500op/s
Request duration reports for insecure-bank
gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section baseline
no_agent (4.226 ms) : 4177, 4275
. : milestone, 4226,
iast (9.753 ms) : 9590, 9916
. : milestone, 9753,
iast_FULL (14.012 ms) : 13729, 14296
. : milestone, 14012,
iast_GLOBAL (10.616 ms) : 10426, 10806
. : milestone, 10616,
profiling (8.988 ms) : 8840, 9137
. : milestone, 8988,
tracing (7.7 ms) : 7590, 7811
. : milestone, 7700,
section candidate
no_agent (4.265 ms) : 4209, 4320
. : milestone, 4265,
iast (9.968 ms) : 9801, 10135
. : milestone, 9968,
iast_FULL (14.393 ms) : 14106, 14679
. : milestone, 14393,
iast_GLOBAL (10.359 ms) : 10177, 10540
. : milestone, 10359,
profiling (8.657 ms) : 8527, 8787
. : milestone, 8657,
tracing (7.866 ms) : 7746, 7987
. : milestone, 7866,
Loading
  • baseline results
VariantRequest duration [CI 0.99]Δ no_agent
no_agent4.226 ms [4.177 ms, 4.275 ms]-
iast9.753 ms [9.59 ms, 9.916 ms]5.527 ms (130.8%)
iast_FULL14.012 ms [13.729 ms, 14.296 ms]9.786 ms (231.6%)
iast_GLOBAL10.616 ms [10.426 ms, 10.806 ms]6.39 ms (151.2%)
profiling8.988 ms [8.84 ms, 9.137 ms]4.762 ms (112.7%)
tracing7.7 ms [7.59 ms, 7.811 ms]3.474 ms (82.2%)
  • candidate results
VariantRequest duration [CI 0.99]Δ no_agent
no_agent4.265 ms [4.209 ms, 4.32 ms]-
iast9.968 ms [9.801 ms, 10.135 ms]5.704 ms (133.7%)
iast_FULL14.393 ms [14.106 ms, 14.679 ms]10.128 ms (237.5%)
iast_GLOBAL10.359 ms [10.177 ms, 10.54 ms]6.094 ms (142.9%)
profiling8.657 ms [8.527 ms, 8.787 ms]4.393 ms (103.0%)
tracing7.866 ms [7.746 ms, 7.987 ms]3.602 ms (84.5%)
Request duration reports for petclinic
gantt
title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section baseline
no_agent (36.804 ms) : 36513, 37096
. : milestone, 36804,
appsec (49.015 ms) : 48572, 49459
. : milestone, 49015,
code_origins (45.454 ms) : 45047, 45861
. : milestone, 45454,
iast (45.002 ms) : 44628, 45376
. : milestone, 45002,
profiling (48.763 ms) : 48317, 49209
. : milestone, 48763,
tracing (44.266 ms) : 43889, 44644
. : milestone, 44266,
section candidate
no_agent (35.572 ms) : 35295, 35850
. : milestone, 35572,
appsec (47.493 ms) : 47059, 47926
. : milestone, 47493,
code_origins (43.277 ms) : 42896, 43657
. : milestone, 43277,
iast (43.825 ms) : 43443, 44208
. : milestone, 43825,
profiling (48.413 ms) : 47960, 48866
. : milestone, 48413,
tracing (43.587 ms) : 43219, 43954
. : milestone, 43587,
Loading
  • baseline results
VariantRequest duration [CI 0.99]Δ no_agent
no_agent36.804 ms [36.513 ms, 37.096 ms]-
appsec49.015 ms [48.572 ms, 49.459 ms]12.211 ms (33.2%)
code_origins45.454 ms [45.047 ms, 45.861 ms]8.649 ms (23.5%)
iast45.002 ms [44.628 ms, 45.376 ms]8.198 ms (22.3%)
profiling48.763 ms [48.317 ms, 49.209 ms]11.959 ms (32.5%)
tracing44.266 ms [43.889 ms, 44.644 ms]7.462 ms (20.3%)
  • candidate results
VariantRequest duration [CI 0.99]Δ no_agent
no_agent35.572 ms [35.295 ms, 35.85 ms]-
appsec47.493 ms [47.059 ms, 47.926 ms]11.92 ms (33.5%)
code_origins43.277 ms [42.896 ms, 43.657 ms]7.705 ms (21.7%)
iast43.825 ms [43.443 ms, 44.208 ms]8.253 ms (23.2%)
profiling48.413 ms [47.96 ms, 48.866 ms]12.84 ms (36.1%)
tracing43.587 ms [43.219 ms, 43.954 ms]8.014 ms (22.5%)

Dacapo

Parameters

BaselineCandidate
baseline_or_candidatebaselinecandidate
git_branchmasterdougqh/spanbuilder-pooling
git_commit_date17611562281761157731
git_commit_shacd02b0c3d9c527
release_version1.55.0-SNAPSHOT~cd02b0c3261.54.0-SNAPSHOT~3d9c52713f
See matching parameters
BaselineCandidate
applicationbiojavabiojava
ci_job_date17611597841761159784
ci_job_id11924863131192486313
ci_pipeline_id8000863680008636
cpu_modelIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHzIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_versionLinux runner-zfyrx7zua-project-304-concurrent-0-am3amwy5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/LinuxLinux runner-zfyrx7zua-project-304-concurrent-0-am3amwy5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava
gantt
title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section baseline
no_agent (15.615 s) : 15615000, 15615000
. : milestone, 15615000,
appsec (15.16 s) : 15160000, 15160000
. : milestone, 15160000,
iast (18.59 s) : 18590000, 18590000
. : milestone, 18590000,
iast_GLOBAL (18.08 s) : 18080000, 18080000
. : milestone, 18080000,
profiling (15.458 s) : 15458000, 15458000
. : milestone, 15458000,
tracing (15.064 s) : 15064000, 15064000
. : milestone, 15064000,
section candidate
no_agent (15.664 s) : 15664000, 15664000
. : milestone, 15664000,
appsec (15.06 s) : 15060000, 15060000
. : milestone, 15060000,
iast (18.512 s) : 18512000, 18512000
. : milestone, 18512000,
iast_GLOBAL (18.238 s) : 18238000, 18238000
. : milestone, 18238000,
profiling (15.227 s) : 15227000, 15227000
. : milestone, 15227000,
tracing (15.033 s) : 15033000, 15033000
. : milestone, 15033000,
Loading
  • baseline results
VariantExecution Time [CI 0.99]Δ no_agent
no_agent15.615 s [15.615 s, 15.615 s]-
appsec15.16 s [15.16 s, 15.16 s]-455.0 ms (-2.9%)
iast18.59 s [18.59 s, 18.59 s]2.975 s (19.1%)
iast_GLOBAL18.08 s [18.08 s, 18.08 s]2.465 s (15.8%)
profiling15.458 s [15.458 s, 15.458 s]-157.0 ms (-1.0%)
tracing15.064 s [15.064 s, 15.064 s]-551.0 ms (-3.5%)
  • candidate results
VariantExecution Time [CI 0.99]Δ no_agent
no_agent15.664 s [15.664 s, 15.664 s]-
appsec15.06 s [15.06 s, 15.06 s]-604.0 ms (-3.9%)
iast18.512 s [18.512 s, 18.512 s]2.848 s (18.2%)
iast_GLOBAL18.238 s [18.238 s, 18.238 s]2.574 s (16.4%)
profiling15.227 s [15.227 s, 15.227 s]-437.0 ms (-2.8%)
tracing15.033 s [15.033 s, 15.033 s]-631.0 ms (-4.0%)
Execution time for tomcat
gantt
title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~3d9c52713f, baseline=1.55.0-SNAPSHOT~cd02b0c326
dateFormat X
axisFormat %s
section baseline
no_agent (1.477 ms) : 1465, 1488
. : milestone, 1477,
appsec (3.75 ms) : 3530, 3969
. : milestone, 3750,
iast (2.215 ms) : 2151, 2279
. : milestone, 2215,
iast_GLOBAL (2.274 ms) : 2209, 2338
. : milestone, 2274,
profiling (2.064 ms) : 2012, 2115
. : milestone, 2064,
tracing (2.037 ms) : 1987, 2087
. : milestone, 2037,
section candidate
no_agent (1.484 ms) : 1472, 1496
. : milestone, 1484,
appsec (3.728 ms) : 3508, 3947
. : milestone, 3728,
iast (2.226 ms) : 2162, 2290
. : milestone, 2226,
iast_GLOBAL (2.263 ms) : 2199, 2327
. : milestone, 2263,
profiling (2.088 ms) : 2035, 2142
. : milestone, 2088,
tracing (2.038 ms) : 1988, 2088
. : milestone, 2038,
Loading
  • baseline results
VariantExecution Time [CI 0.99]Δ no_agent
no_agent1.477 ms [1.465 ms, 1.488 ms]-
appsec3.75 ms [3.53 ms, 3.969 ms]2.273 ms (153.9%)
iast2.215 ms [2.151 ms, 2.279 ms]738.525 µs (50.0%)
iast_GLOBAL2.274 ms [2.209 ms, 2.338 ms]796.939 µs (54.0%)
profiling2.064 ms [2.012 ms, 2.115 ms]586.987 µs (39.8%)
tracing2.037 ms [1.987 ms, 2.087 ms]560.23 µs (37.9%)
  • candidate results
VariantExecution Time [CI 0.99]Δ no_agent
no_agent1.484 ms [1.472 ms, 1.496 ms]-
appsec3.728 ms [3.508 ms, 3.947 ms]2.243 ms (151.2%)
iast2.226 ms [2.162 ms, 2.29 ms]741.441 µs (50.0%)
iast_GLOBAL2.263 ms [2.199 ms, 2.327 ms]778.77 µs (52.5%)
profiling2.088 ms [2.035 ms, 2.142 ms]603.984 µs (40.7%)
tracing2.038 ms [1.988 ms, 2.088 ms]553.539 µs (37.3%)

public static final String OPTIMIZED_MAP_ENABLED = "optimized.map.enabled";
public static final String TAG_NAME_UTF8_CACHE_SIZE = "tag.name.utf8.cache.size";
public static final String TAG_VALUE_UTF8_CACHE_SIZE = "tag.value.utf8.cache.size";
public static final String SPAN_BUILDER_REUSE_ENABLED = "span.builder.reuse.enabled";

@dougqhdougqhSep 15, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is best class to be placing these in. Or if this is the proper namespace. dd.trace... might be better

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree dd.trace prefix fits better

@PerfectSlayerPerfectSlayer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ question: ‏Quick question for my understanding: how is that supposed to work with manual tracing API? (DD or OTel)

Let’s say:

varspanBuilder = tracer.spanBuilder("span1");
// …varspan = tracer.spanBuilder("span2").startSpan();
// …spanBuilder.setAttribute("key", "value").startSpan(); 

The second startSpan() call will start the span with the wrong name ("span2" instead of "span1") and there is nothing we can do on the API to prevent such use case.

@dougqh

dougqh commented Sep 16, 2025

Copy link
Copy Markdown
ContributorAuthor

The second startSpan() call will start the span with the wrong name ("span2" instead of "span1") and there is nothing we can do on the API to prevent such use case.
Actually, I updated my approach yesterday to handle that specific case; however, other cases are definitely worth considering.

The solution that I put in place yesterday was that I track whether or not the SpanBuilder is "in-use". A SpanBuilder is considered "in -use" from the point that it is reset during CoreTracer.buildSpan up until a span is created with SpanBuilder.buildSpan.

So CoreTracer.buildSpan can check if the ThreadLocal SpanBuilder is still "in-use". If it is, then the CoreTracer still constructs a new SpanBuilder and returns that instead.

So the 'in-use' check solves the case mentioned in the original comment. You'll still get two distinct SpanBuilders.
I definitely should add a test for that. Right now, I just wanted a sanity check from others and the automated benchmarking.

But there are still other cases that worry me, specifically, I'm worried that you can actually call build on the SpanBuilder multiple times. That's not idiomatic, but it would probably work today. My solution doesn't handle that.

I don't believe our own instrumentation produces multiple Spans per builder today, so we could solve the problem by having two slightly different sets of rules: one for automatic and one for manual. We can accomplish that by having two sets of methods: one set that will reuse SpanBuilders that we use in automatic instrumentation, and another set that always creates a new SpanBuilder that we use in manual instrumentation (and the OTel bridge).

UPDATE:
I realized that there was an easy middle ground with builder reuse issues.

buildSpan can maintain the existing semantics including allowing building multiple spans.
The startSpan convenience methods can safely reuse a SpanBuilder, since we know that it only builds a single span.
And to do that, I introduced a new singleSpanBuilder method that we can use in automatic instrumentation. singleSpanBuilder can use SpanBuilder recycling under the covers as long as we know it is not "in-use" (see above)

That does mean that we have to update some instrumentation to get the full benefit, but at least, we know the change is safe.

Refactored code, so tests work regardless of Config
To avoid breaking any potential code that builds multiple spans from the same SpanBuilder, updated the SpanBuilder pooling approach
Introduced a new method singleSpanBuilder which can build one and only one span, this method can be used by automatic instrumentation as an optimization.
singleSpanBuilder is now used inside the startSpan convenience methods, since we know they only build and return one span. Any automatic instrumentation using startSpan gets the optimization for free.
buildSpan maintains its original semantics, so all existing continues to work as is.
@dougqh
dougqh requested a review from a team as a code ownerSeptember 16, 2025 20:11
In a microbenchmark, buildSpan was performing worse than previously.
To address, that shortcoming and to clean-up the code...
Made CoreSpanBuilder abstract and introduced two child classes: MultiSpanBuilder and ReusableSingleSpanBuilder
MultiSpanBuilder is used by buildSpan
ReusableSingleSpanBuilder is used by singleSpanBuilder / startSpan (indirectly)

@AlexeyKuznetsov-DDAlexeyKuznetsov-DD left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just left several minor comments.

Comment on lines +88 to +105
MethodHandle h_startVThread;
try {
h_startVThread =
MethodHandles.lookup()
.findStatic(
Thread.class,
"startVirtualThread",
MethodType.methodType(Thread.class, Runnable.class));
} catch (NoSuchMethodException | IllegalAccessException e) {
throw new IllegalStateException(e);
}

try {
return (Thread) h_startVThread.invoke(runnable);
} catch (Throwable e) {
throw new IllegalStateException(e);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a very minor nitpick: I guess we can simplify code by removing try/catch?
We already checking for V-threads before calling this function.


@Override
public CoreSpanBuilder withTag(final String tag, final Number number) {
public final CoreSpanBuilder withTag(final String tag, final Number number) {

@AlexeyKuznetsov-DDAlexeyKuznetsov-DDOct 17, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still curious why we are using final so often? It will force Java compiler to produce more optimal code?

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor

Also noticed in build logs on CI:

branches covered ratio is 0.3, but expected minimum is 0.5

@dougqh

dougqh commented Oct 22, 2025

Copy link
Copy Markdown
ContributorAuthor

Also noticed in build logs on CI:

branches covered ratio is 0.3, but expected minimum is 0.5

Yes, that's what I've been trying to figure out. I think it might have been an issue with my creating a CoreTracerTest.java in addition to CoreTracerTest.groovy.

@dougqh
dougqh merged commit 4c89f79 into masterOct 22, 2025
533 of 536 checks passed
@dougqh
dougqh deleted the dougqh/spanbuilder-pooling branch October 22, 2025 19:18
@github-actionsgithub-actionsBot added this to the 1.55.0 milestone Oct 22, 2025
@bric3bric3 added the tag: concurrency Virtual Threads, Coroutines, Async, RX, Executors label Feb 2, 2026
gh-worker-dd-mergequeue-cf854dBot pushed a commit that referenced this pull request Jul 22, 2026
Add front-half span-creation JMH benchmarks
Adds application-thread (front-half) allocation benchmarks that ground the
TagMap 2.0 / SpanPrototype work against the real span lifecycle:
- SpanCreationBenchmark (front-A): single span create -> tag -> finish. Bare
baseline (startSpan vs buildSpan), two known-tag shapes set after start —
web-server (7 tags) and JDBC/DB client (9 tags) — plus a builder-tag-path arm
(withTag before start, the OTel-bridge shape) to track how the startSpan /
buildSpan lineages diverge across releases.
- TraceAssemblyBenchmark (front-B): web-shape root + N children (childCount
1/5/20), exercising the per-span baseline-tag copy that map-to-map copy
(TagMap 1.0) and the trace/span tag split (level-split) target.
- DropWriter: shared no-op Writer so finish() excludes serialization / agent
I/O, isolating front-half allocation for -prof gc.
Deliberately drift-stable (v1.53->master API) so the same file can be grafted
onto old release tags for a historical allocation curve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add virtual-thread span-creation benchmark
Runs startSpan create+finish on a virtual thread — the regime the platform-thread
SpanCreationBenchmark is blind to. startSpan's thread-local SpanBuilder reuse
(#9537) is disabled on virtual threads, so a builder is allocated per span there;
the builder bypass (#9998) removes it. This bench is where that difference shows.
Starts the virtual thread via reflection (Thread.startVirtualThread) so the jmh
source set still compiles on pre-21 toolchains; requires a 21+ JDK at run time.
The per-op vthread spawn/join cost is constant across tracer versions, so it
cancels in the version-over-version delta.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document historical allocation/throughput in benchmark headers
Fold the v1.53.0 -> 1.64.0 sweep results into the SpanCreationBenchmark and
TraceAssemblyBenchmark javadoc so the trend is discoverable from the source
without re-running. Allocation (gc.alloc.rate.norm B/op) carries a net Δ%;
throughput (ops/us) is included but flagged directional-only and given no Δ%
since it has no reliable trend (laptop thermals + per-fork inlining bimodality).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merge branch 'master' into dougqh/span-creation-bench
Give the jdbc-shaped benchmark span its own operation name
A real jdbc span is never "servlet.request"; using a distinct
"database.query" operation name keeps the web- and DB-shaped span
benchmarks from being conflated by operation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename OPERATION_NAME to SERVER_OPERATION_NAME in the span benchmark
Now that the DB-shaped span has its own JDBC_OPERATION_NAME, name the
web/server operation constant symmetrically for clarity.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use a Blackhole in DropWriter and force WARN logging in span-creation benchmarks
DropWriter.write() was a true no-op, letting the JIT treat the
finish()-triggered write as dead code; hand the trace to a Blackhole
instead. Also force -DTEST_LOG_LEVEL=warn on all three benchmark
classes' forks, since logback.xml defaults to DEBUG and the benchmarks'
own javadoc requires WARN to keep DEBUG logging from corrupting the
-prof gc allocation numbers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merge branch 'master' into dougqh/span-creation-bench
Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: coreTracer coretag: concurrencyVirtual Threads, Coroutines, Async, RX, Executorstag: performancePerformance related changestype: featureEnhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@dougqh@PerfectSlayer@AlexeyKuznetsov-DD@bric3