R&D week: blackhole span by amarziali · Pull Request #6326 · DataDog/dd-trace-java

amarziali · 2023-12-06T16:50:13Z

What Does This Do

Introduces the blackhole span who's muting the tracing during its lifetime (as soon as a blackhole scope is opened)
As analogy with the @Trace annotation, a @DoNotTrace annotation is also provided in order to decorate a method with a blackhole activated span.

It also provides the possibility to obtain a blackhole scope using the DD public API.

Finally showcases how it can be used in an instrumentation. In this case the example is in decoupling the apache http client instrumentation from the aws sdk one

Motivation

Additional Notes

Jira ticket: [PROJ-IDENT]

pr-commenter · 2023-12-06T17:32:11Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/blackhole
git_commit_date	1704706030	1704739464
git_commit_sha	`5bf1b2f`	`1105e7f`
release_version	1.28.0-SNAPSHOT~5bf1b2f68e	1.28.0-SNAPSHOT~1105e7f502

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1704742062	1704742062
ci_job_id	403145314	403145314
ci_pipeline_id	26067995	26067995
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 9 unstable metrics.

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049263
Total [baseline] (8.71 s) : 0, 8710407
Agent [candidate] (1.054 s) : 0, 1053972
Total [candidate] (8.712 s) : 0, 8711754
section iast
Agent [baseline] (1.168 s) : 0, 1167713
Total [baseline] (9.234 s) : 0, 9234058
Agent [candidate] (1.18 s) : 0, 1180195
Total [candidate] (9.278 s) : 0, 9278168
section iast_TELEMETRY_OFF
Agent [baseline] (1.172 s) : 0, 1171895
Total [baseline] (9.248 s) : 0, 9248266
Agent [candidate] (1.171 s) : 0, 1170553
Total [candidate] (9.247 s) : 0, 9247283

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	iast	1.168 s	118.451 ms (11.3%)
Agent	iast_TELEMETRY_OFF	1.172 s	122.633 ms (11.7%)
Total	tracing	8.71 s	-
Total	iast	9.234 s	523.651 ms (6.0%)
Total	iast_TELEMETRY_OFF	9.248 s	537.859 ms (6.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	iast	1.18 s	126.223 ms (12.0%)
Agent	iast_TELEMETRY_OFF	1.171 s	116.581 ms (11.1%)
Total	tracing	8.712 s	-
Total	iast	9.278 s	566.414 ms (6.5%)
Total	iast_TELEMETRY_OFF	9.247 s	535.529 ms (6.1%)

gantt
title insecure-bank - break down per module: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (649.595 ms) : 0, 649595
BytebuddyAgent [candidate] (652.112 ms) : 0, 652112
GlobalTracer [baseline] (306.709 ms) : 0, 306709
GlobalTracer [candidate] (308.864 ms) : 0, 308864
AppSec [baseline] (50.865 ms) : 0, 50865
AppSec [candidate] (50.697 ms) : 0, 50697
Remote Config [baseline] (672.161 µs) : 0, 672
Remote Config [candidate] (674.925 µs) : 0, 675
Telemetry [baseline] (7.221 ms) : 0, 7221
Telemetry [candidate] (7.239 ms) : 0, 7239
section iast
BytebuddyAgent [baseline] (769.717 ms) : 0, 769717
BytebuddyAgent [candidate] (777.459 ms) : 0, 777459
GlobalTracer [baseline] (284.642 ms) : 0, 284642
GlobalTracer [candidate] (288.102 ms) : 0, 288102
AppSec [baseline] (51.439 ms) : 0, 51439
AppSec [candidate] (53.726 ms) : 0, 53726
IAST [baseline] (20.65 ms) : 0, 20650
IAST [candidate] (19.241 ms) : 0, 19241
Remote Config [baseline] (567.423 µs) : 0, 567
Remote Config [candidate] (567.665 µs) : 0, 568
Telemetry [baseline] (6.441 ms) : 0, 6441
Telemetry [candidate] (6.467 ms) : 0, 6467
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (769.395 ms) : 0, 769395
BytebuddyAgent [candidate] (768.745 ms) : 0, 768745
GlobalTracer [baseline] (288.078 ms) : 0, 288078
GlobalTracer [candidate] (287.579 ms) : 0, 287579
AppSec [baseline] (49.81 ms) : 0, 49810
AppSec [candidate] (49.462 ms) : 0, 49462
IAST [baseline] (22.96 ms) : 0, 22960
IAST [candidate] (23.34 ms) : 0, 23340
Remote Config [baseline] (631.682 µs) : 0, 632
Remote Config [candidate] (603.368 µs) : 0, 603
Telemetry [baseline] (6.463 ms) : 0, 6463
Telemetry [candidate] (6.392 ms) : 0, 6392

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053330
Total [baseline] (9.374 s) : 0, 9373900
Agent [candidate] (1.059 s) : 0, 1059420
Total [candidate] (9.36 s) : 0, 9360284
section appsec
Agent [baseline] (1.148 s) : 0, 1148194
Total [baseline] (9.447 s) : 0, 9447453
Agent [candidate] (1.15 s) : 0, 1149684
Total [candidate] (9.442 s) : 0, 9442355
section iast
Agent [baseline] (1.178 s) : 0, 1178055
Total [baseline] (9.583 s) : 0, 9583103
Agent [candidate] (1.179 s) : 0, 1179318
Total [candidate] (9.586 s) : 0, 9585596
section profiling
Agent [baseline] (1.273 s) : 0, 1273418
Total [baseline] (9.631 s) : 0, 9630822
Agent [candidate] (1.281 s) : 0, 1280666
Total [candidate] (9.689 s) : 0, 9689076

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	appsec	1.148 s	94.863 ms (9.0%)
Agent	iast	1.178 s	124.725 ms (11.8%)
Agent	profiling	1.273 s	220.088 ms (20.9%)
Total	tracing	9.374 s	-
Total	appsec	9.447 s	73.553 ms (0.8%)
Total	iast	9.583 s	209.203 ms (2.2%)
Total	profiling	9.631 s	256.922 ms (2.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.059 s	-
Agent	appsec	1.15 s	90.264 ms (8.5%)
Agent	iast	1.179 s	119.898 ms (11.3%)
Agent	profiling	1.281 s	221.246 ms (20.9%)
Total	tracing	9.36 s	-
Total	appsec	9.442 s	82.071 ms (0.9%)
Total	iast	9.586 s	225.312 ms (2.4%)
Total	profiling	9.689 s	328.792 ms (3.5%)

gantt
title petclinic - break down per module: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (653.172 ms) : 0, 653172
BytebuddyAgent [candidate] (656.389 ms) : 0, 656389
GlobalTracer [baseline] (307.108 ms) : 0, 307108
GlobalTracer [candidate] (309.631 ms) : 0, 309631
AppSec [baseline] (50.658 ms) : 0, 50658
AppSec [candidate] (50.761 ms) : 0, 50761
Remote Config [baseline] (674.566 µs) : 0, 675
Remote Config [candidate] (689.576 µs) : 0, 690
Telemetry [baseline] (7.237 ms) : 0, 7237
Telemetry [candidate] (7.208 ms) : 0, 7208
section appsec
BytebuddyAgent [baseline] (650.373 ms) : 0, 650373
BytebuddyAgent [candidate] (650.871 ms) : 0, 650871
GlobalTracer [baseline] (307.526 ms) : 0, 307526
GlobalTracer [candidate] (308.422 ms) : 0, 308422
AppSec [baseline] (148.523 ms) : 0, 148523
AppSec [candidate] (148.579 ms) : 0, 148579
Remote Config [baseline] (641.231 µs) : 0, 641
Remote Config [candidate] (645.457 µs) : 0, 645
Telemetry [baseline] (6.891 ms) : 0, 6891
Telemetry [candidate] (6.852 ms) : 0, 6852
section iast
BytebuddyAgent [baseline] (777.722 ms) : 0, 777722
BytebuddyAgent [candidate] (777.723 ms) : 0, 777723
GlobalTracer [baseline] (286.026 ms) : 0, 286026
GlobalTracer [candidate] (288.337 ms) : 0, 288337
AppSec [baseline] (53.497 ms) : 0, 53497
AppSec [candidate] (53.4 ms) : 0, 53400
IAST [baseline] (18.349 ms) : 0, 18349
IAST [candidate] (17.892 ms) : 0, 17892
Remote Config [baseline] (586.205 µs) : 0, 586
Remote Config [candidate] (578.672 µs) : 0, 579
Telemetry [baseline] (7.275 ms) : 0, 7275
Telemetry [candidate] (6.589 ms) : 0, 6589
section profiling
BytebuddyAgent [baseline] (660.359 ms) : 0, 660359
BytebuddyAgent [candidate] (665.683 ms) : 0, 665683
GlobalTracer [baseline] (374.906 ms) : 0, 374906
GlobalTracer [candidate] (376.609 ms) : 0, 376609
AppSec [baseline] (51.236 ms) : 0, 51236
AppSec [candidate] (51.475 ms) : 0, 51475
Remote Config [baseline] (1.014 ms) : 0, 1014
Remote Config [candidate] (1.002 ms) : 0, 1002
Telemetry [baseline] (7.248 ms) : 0, 7248
Telemetry [candidate] (7.361 ms) : 0, 7361
ProfilingAgent [baseline] (124.319 ms) : 0, 124319
ProfilingAgent [candidate] (123.84 ms) : 0, 123840
Profiling [baseline] (124.344 ms) : 0, 124344
Profiling [candidate] (123.865 ms) : 0, 123865

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-01-08T19:06:54	2024-01-08T19:23:28
git_branch	master	andrea.marziali/blackhole
git_commit_date	1704706030	1704739464
git_commit_sha	`5bf1b2f`	`1105e7f`
release_version	1.28.0-SNAPSHOT~5bf1b2f68e	1.28.0-SNAPSHOT~1105e7f502
start_time	2024-01-08T19:06:41	2024-01-08T19:23:15

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1704742062	1704742062
ci_job_id	403145314	403145314
ci_pipeline_id	26067995	26067995
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 13 unstable metrics.

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
dateFormat X
axisFormat %s
section baseline
no_agent (359.35 µs) : 340, 379
. : milestone, 359,
iast (476.785 µs) : 456, 497
. : milestone, 477,
iast_FULL (545.864 µs) : 525, 566
. : milestone, 546,
iast_INACTIVE (448.723 µs) : 428, 470
. : milestone, 449,
iast_TELEMETRY_OFF (464.861 µs) : 444, 485
. : milestone, 465,
tracing (437.77 µs) : 417, 459
. : milestone, 438,
section candidate
no_agent (363.367 µs) : 343, 384
. : milestone, 363,
iast (470.084 µs) : 450, 491
. : milestone, 470,
iast_FULL (536.006 µs) : 516, 556
. : milestone, 536,
iast_INACTIVE (440.044 µs) : 420, 460
. : milestone, 440,
iast_TELEMETRY_OFF (468.398 µs) : 447, 490
. : milestone, 468,
tracing (442.466 µs) : 422, 463
. : milestone, 442,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	359.35 µs [339.706 µs, 378.994 µs]	-
iast	476.785 µs [456.164 µs, 497.405 µs]	117.435 µs (32.7%)
iast_FULL	545.864 µs [525.486 µs, 566.241 µs]	186.514 µs (51.9%)
iast_INACTIVE	448.723 µs [427.839 µs, 469.607 µs]	89.373 µs (24.9%)
iast_TELEMETRY_OFF	464.861 µs [444.389 µs, 485.333 µs]	105.511 µs (29.4%)
tracing	437.77 µs [416.606 µs, 458.934 µs]	78.42 µs (21.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	363.367 µs [342.854 µs, 383.879 µs]	-
iast	470.084 µs [449.609 µs, 490.56 µs]	106.718 µs (29.4%)
iast_FULL	536.006 µs [515.693 µs, 556.318 µs]	172.639 µs (47.5%)
iast_INACTIVE	440.044 µs [419.808 µs, 460.281 µs]	76.677 µs (21.1%)
iast_TELEMETRY_OFF	468.398 µs [447.204 µs, 489.593 µs]	105.032 µs (28.9%)
tracing	442.466 µs [421.506 µs, 463.426 µs]	79.099 µs (21.8%)

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
dateFormat X
axisFormat %s
section baseline
no_agent (1.344 ms) : 1325, 1363
. : milestone, 1344,
appsec (1.758 ms) : 1732, 1783
. : milestone, 1758,
iast (1.525 ms) : 1501, 1549
. : milestone, 1525,
profiling (1.552 ms) : 1526, 1578
. : milestone, 1552,
tracing (1.484 ms) : 1459, 1509
. : milestone, 1484,
section candidate
no_agent (1.342 ms) : 1323, 1361
. : milestone, 1342,
appsec (1.772 ms) : 1747, 1797
. : milestone, 1772,
iast (1.526 ms) : 1502, 1550
. : milestone, 1526,
profiling (1.519 ms) : 1494, 1544
. : milestone, 1519,
tracing (1.501 ms) : 1476, 1526
. : milestone, 1501,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.344 ms [1.325 ms, 1.363 ms]	-
appsec	1.758 ms [1.732 ms, 1.783 ms]	413.788 µs (30.8%)
iast	1.525 ms [1.501 ms, 1.549 ms]	181.316 µs (13.5%)
profiling	1.552 ms [1.526 ms, 1.578 ms]	207.912 µs (15.5%)
tracing	1.484 ms [1.459 ms, 1.509 ms]	140.244 µs (10.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.342 ms [1.323 ms, 1.361 ms]	-
appsec	1.772 ms [1.747 ms, 1.797 ms]	430.09 µs (32.0%)
iast	1.526 ms [1.502 ms, 1.55 ms]	184.002 µs (13.7%)
profiling	1.519 ms [1.494 ms, 1.544 ms]	177.092 µs (13.2%)
tracing	1.501 ms [1.476 ms, 1.526 ms]	159.164 µs (11.9%)

ygree

Looks good to me

The tracer sometimes runs external processes in some cases. Currently, `hostname` and `getenforce`. The subprocess instrumentation creates spans for these, leading to visible `command_execution` spans for `hostname` and/or `enforce` which may be confusing to users. This commit uses the bew blackhole span API (#6326) to mute tracing during when `hostname` and `getenforce` are run, preventing the creation of these spans.

amarziali force-pushed the andrea.marziali/blackhole branch 2 times, most recently from 1b07497 to 6628dbbCompare December 8, 2023 13:23

amarziali marked this pull request as ready for review December 13, 2023 14:13

amarziali requested a review from a team as a code owner December 13, 2023 14:13

amarziali requested review from mcculls and ygree December 13, 2023 14:13

amarziali added comp: core Tracer core tag: experimental Experimental changes comp: api Tracer public API inst: trace annotation Trace annotations instrumentation comp: opentracing OpenTracing labels Dec 13, 2023

ygree reviewed Jan 3, 2024
View reviewed changes

Comment threadinternal-api/src/main/java/datadog/trace/bootstrap/instrumentation/api/AgentTracer.java

amarziali added 4 commits January 5, 2024 11:46

R&D week: blackhole span
37ae5c2

blackhole manual tracing API
af0cddb

Decouple httpclient and aws sdk instrumentations with blackhole
25b195a

exclude blackhole from coverage
5508303

amarziali force-pushed the andrea.marziali/blackhole branch from 4964cb5 to 5508303Compare January 5, 2024 10:52

amarziali requested a review from ygree January 5, 2024 10:54

ygree reviewed Jan 5, 2024
View reviewed changes

Comment thread...httpclient-4/src/main/java/datadog/trace/instrumentation/apachehttpclient/HelperMethods.java

ygree approved these changes Jan 5, 2024
View reviewed changes

Merge branch 'master' into andrea.marziali/blackhole
f5f7da5

amarziali enabled auto-merge (squash) January 8, 2024 18:44

amarziali added 2 commits January 8, 2024 19:44

Merge branch 'master' into andrea.marziali/blackhole
1105e7f

exlcude no-op from coverage
47f4209

amarziali merged commit f46fcc6 into masterJan 9, 2024

amarziali deleted the andrea.marziali/blackhole branch January 9, 2024 11:06

github-actionsBot added this to the 1.28.0 milestone Jan 9, 2024

smola mentioned this pull request Jan 30, 2024
Mute tracing for subprocesses executed in by the tracer #6584
Merged

Navigation Menu

Uh oh!

R&D week: blackhole span - #6326

R&D week: blackhole span#6326
amarziali merged 7 commits into
masterfrom
andrea.marziali/blackhole

amarziali commented Dec 6, 2023•
edited
Loading

Uh oh!

pr-commenterBot commented Dec 6, 2023•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

ygree left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

amarziali commented Dec 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenterBot commented Dec 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Uh oh!

Uh oh!

ygree left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

amarziali commented Dec 6, 2023•
edited
Loading

pr-commenterBot commented Dec 6, 2023•
edited
Loading