Don't create a process span for agent internal threads by bantonsson · Pull Request #6478 · DataDog/dd-trace-java

bantonsson · 2024-01-12T12:53:41Z

What Does This Do

Skips creating a process span for process execution that is initialized from one of the agent internal threads.

Motivation

Internal calls to hostname and other commands from the agent during setup runs at various times and pollute smoke tests with unrelated traces.

pr-commenter · 2024-01-12T13:36:55Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	ban/no-process-spans-for-agent
git_commit_date	1705056379	1705064087
git_commit_sha	`656c660`	`39a26b8`
release_version	1.28.0-SNAPSHOT~656c6600e0	1.28.0-SNAPSHOT~39a26b859c

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1705066720	1705066720
ci_job_id	407067830	407067830
ci_pipeline_id	26392532	26392532
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 44 metrics, 10 unstable metrics.

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.28.0-SNAPSHOT~39a26b859c, baseline=1.28.0-SNAPSHOT~656c6600e0
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1051394
Total [baseline] (8.713 s) : 0, 8712514
Agent [candidate] (1.053 s) : 0, 1053054
Total [candidate] (8.728 s) : 0, 8728300
section iast
Agent [baseline] (1.178 s) : 0, 1178007
Total [baseline] (9.242 s) : 0, 9242044
Agent [candidate] (1.181 s) : 0, 1180692
Total [candidate] (9.239 s) : 0, 9238761
section iast_TELEMETRY_OFF
Agent [baseline] (1.178 s) : 0, 1177601
Total [baseline] (9.278 s) : 0, 9278180
Agent [candidate] (1.169 s) : 0, 1169078
Total [candidate] (9.282 s) : 0, 9282364

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	iast	1.178 s	126.613 ms (12.0%)
Agent	iast_TELEMETRY_OFF	1.178 s	126.208 ms (12.0%)
Total	tracing	8.713 s	-
Total	iast	9.242 s	529.529 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.278 s	565.665 ms (6.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	iast	1.181 s	127.639 ms (12.1%)
Agent	iast_TELEMETRY_OFF	1.169 s	116.024 ms (11.0%)
Total	tracing	8.728 s	-
Total	iast	9.239 s	510.46 ms (5.8%)
Total	iast_TELEMETRY_OFF	9.282 s	554.063 ms (6.3%)

gantt
title insecure-bank - break down per module: candidate=1.28.0-SNAPSHOT~39a26b859c, baseline=1.28.0-SNAPSHOT~656c6600e0
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (664.187 ms) : 0, 664187
BytebuddyAgent [candidate] (665.401 ms) : 0, 665401
GlobalTracer [baseline] (294.748 ms) : 0, 294748
GlobalTracer [candidate] (295.051 ms) : 0, 295051
AppSec [baseline] (50.467 ms) : 0, 50467
AppSec [candidate] (50.628 ms) : 0, 50628
Remote Config [baseline] (657.349 µs) : 0, 657
Remote Config [candidate] (645.223 µs) : 0, 645
Telemetry [baseline] (7.184 ms) : 0, 7184
Telemetry [candidate] (7.156 ms) : 0, 7156
section iast
BytebuddyAgent [baseline] (776.73 ms) : 0, 776730
BytebuddyAgent [candidate] (778.358 ms) : 0, 778358
GlobalTracer [baseline] (285.494 ms) : 0, 285494
GlobalTracer [candidate] (287.286 ms) : 0, 287286
AppSec [baseline] (52.807 ms) : 0, 52807
AppSec [candidate] (53.428 ms) : 0, 53428
IAST [baseline] (20.714 ms) : 0, 20714
IAST [candidate] (19.962 ms) : 0, 19962
Remote Config [baseline] (601.569 µs) : 0, 602
Remote Config [candidate] (599.232 µs) : 0, 599
Telemetry [baseline] (7.26 ms) : 0, 7260
Telemetry [candidate] (6.575 ms) : 0, 6575
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (773.89 ms) : 0, 773890
BytebuddyAgent [candidate] (767.959 ms) : 0, 767959
GlobalTracer [baseline] (287.905 ms) : 0, 287905
GlobalTracer [candidate] (285.952 ms) : 0, 285952
AppSec [baseline] (54.102 ms) : 0, 54102
AppSec [candidate] (53.205 ms) : 0, 53205
IAST [baseline] (20.184 ms) : 0, 20184
IAST [candidate] (20.694 ms) : 0, 20694
Remote Config [baseline] (587.111 µs) : 0, 587
Remote Config [candidate] (583.564 µs) : 0, 584
Telemetry [baseline] (6.387 ms) : 0, 6387
Telemetry [candidate] (6.39 ms) : 0, 6390

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.28.0-SNAPSHOT~39a26b859c, baseline=1.28.0-SNAPSHOT~656c6600e0
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053139
Total [baseline] (9.354 s) : 0, 9353593
Agent [candidate] (1.063 s) : 0, 1062588
Total [candidate] (9.35 s) : 0, 9350312
section appsec
Agent [baseline] (1.152 s) : 0, 1151587
Total [baseline] (9.459 s) : 0, 9458914
Agent [candidate] (1.155 s) : 0, 1154983
Total [candidate] (9.493 s) : 0, 9493391
section iast
Agent [baseline] (1.177 s) : 0, 1176980
Total [baseline] (9.584 s) : 0, 9584326
Agent [candidate] (1.187 s) : 0, 1186743
Total [candidate] (9.644 s) : 0, 9644003
section profiling
Agent [baseline] (1.273 s) : 0, 1273002
Total [baseline] (9.591 s) : 0, 9590642
Agent [candidate] (1.277 s) : 0, 1277381
Total [candidate] (9.584 s) : 0, 9583639

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	appsec	1.152 s	98.448 ms (9.3%)
Agent	iast	1.177 s	123.841 ms (11.8%)
Agent	profiling	1.273 s	219.863 ms (20.9%)
Total	tracing	9.354 s	-
Total	appsec	9.459 s	105.321 ms (1.1%)
Total	iast	9.584 s	230.734 ms (2.5%)
Total	profiling	9.591 s	237.049 ms (2.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.063 s	-
Agent	appsec	1.155 s	92.395 ms (8.7%)
Agent	iast	1.187 s	124.155 ms (11.7%)
Agent	profiling	1.277 s	214.793 ms (20.2%)
Total	tracing	9.35 s	-
Total	appsec	9.493 s	143.078 ms (1.5%)
Total	iast	9.644 s	293.691 ms (3.1%)
Total	profiling	9.584 s	233.327 ms (2.5%)

gantt
title petclinic - break down per module: candidate=1.28.0-SNAPSHOT~39a26b859c, baseline=1.28.0-SNAPSHOT~656c6600e0
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (665.249 ms) : 0, 665249
BytebuddyAgent [candidate] (671.143 ms) : 0, 671143
GlobalTracer [baseline] (295.292 ms) : 0, 295292
GlobalTracer [candidate] (297.816 ms) : 0, 297816
AppSec [baseline] (50.496 ms) : 0, 50496
AppSec [candidate] (51.058 ms) : 0, 51058
Remote Config [baseline] (658.656 µs) : 0, 659
Remote Config [candidate] (665.442 µs) : 0, 665
Telemetry [baseline] (7.199 ms) : 0, 7199
Telemetry [candidate] (7.314 ms) : 0, 7314
section appsec
BytebuddyAgent [baseline] (665.853 ms) : 0, 665853
BytebuddyAgent [candidate] (666.67 ms) : 0, 666670
GlobalTracer [baseline] (295.447 ms) : 0, 295447
GlobalTracer [candidate] (297.241 ms) : 0, 297241
AppSec [baseline] (148.525 ms) : 0, 148525
AppSec [candidate] (149.144 ms) : 0, 149144
Remote Config [baseline] (643.714 µs) : 0, 644
Remote Config [candidate] (646.678 µs) : 0, 647
Telemetry [baseline] (6.858 ms) : 0, 6858
Telemetry [candidate] (6.913 ms) : 0, 6913
section iast
BytebuddyAgent [baseline] (774.462 ms) : 0, 774462
BytebuddyAgent [candidate] (783.225 ms) : 0, 783225
GlobalTracer [baseline] (286.29 ms) : 0, 286290
GlobalTracer [candidate] (287.803 ms) : 0, 287803
AppSec [baseline] (51.645 ms) : 0, 51645
AppSec [candidate] (51.476 ms) : 0, 51476
IAST [baseline] (21.625 ms) : 0, 21625
IAST [candidate] (21.492 ms) : 0, 21492
Remote Config [baseline] (604.015 µs) : 0, 604
Remote Config [candidate] (625.011 µs) : 0, 625
Telemetry [baseline] (8.054 ms) : 0, 8054
Telemetry [candidate] (7.33 ms) : 0, 7330
section profiling
BytebuddyAgent [baseline] (662.419 ms) : 0, 662419
BytebuddyAgent [candidate] (665.391 ms) : 0, 665391
GlobalTracer [baseline] (374.584 ms) : 0, 374584
GlobalTracer [candidate] (376.032 ms) : 0, 376032
AppSec [baseline] (50.929 ms) : 0, 50929
AppSec [candidate] (50.861 ms) : 0, 50861
Remote Config [baseline] (985.261 µs) : 0, 985
Remote Config [candidate] (1.015 ms) : 0, 1015
Telemetry [baseline] (7.206 ms) : 0, 7206
Telemetry [candidate] (7.25 ms) : 0, 7250
ProfilingAgent [baseline] (122.622 ms) : 0, 122622
ProfilingAgent [candidate] (122.384 ms) : 0, 122384
Profiling [baseline] (122.646 ms) : 0, 122646
Profiling [candidate] (122.409 ms) : 0, 122409

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-01-12T13:18:02	2024-01-12T13:34:29
git_branch	master	ban/no-process-spans-for-agent
git_commit_date	1705056379	1705064087
git_commit_sha	`656c660`	`39a26b8`
release_version	1.28.0-SNAPSHOT~656c6600e0	1.28.0-SNAPSHOT~39a26b859c
start_time	2024-01-12T13:17:49	2024-01-12T13:34:16

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1705066720	1705066720
ci_job_id	407067830	407067830
ci_pipeline_id	26392532	26392532
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~39a26b859c, baseline=1.28.0-SNAPSHOT~656c6600e0
dateFormat X
axisFormat %s
section baseline
no_agent (366.102 µs) : 346, 386
. : milestone, 366,
iast (470.853 µs) : 450, 491
. : milestone, 471,
iast_FULL (534.577 µs) : 514, 555
. : milestone, 535,
iast_INACTIVE (442.485 µs) : 422, 463
. : milestone, 442,
iast_TELEMETRY_OFF (467.584 µs) : 447, 488
. : milestone, 468,
tracing (441.406 µs) : 420, 463
. : milestone, 441,
section candidate
no_agent (361.573 µs) : 342, 382
. : milestone, 362,
iast (470.43 µs) : 450, 491
. : milestone, 470,
iast_FULL (533.271 µs) : 513, 554
. : milestone, 533,
iast_INACTIVE (445.973 µs) : 425, 467
. : milestone, 446,
iast_TELEMETRY_OFF (465.799 µs) : 445, 486
. : milestone, 466,
tracing (436.331 µs) : 416, 457
. : milestone, 436,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.102 µs [345.736 µs, 386.468 µs]	-
iast	470.853 µs [450.426 µs, 491.281 µs]	104.751 µs (28.6%)
iast_FULL	534.577 µs [513.89 µs, 555.264 µs]	168.475 µs (46.0%)
iast_INACTIVE	442.485 µs [421.738 µs, 463.233 µs]	76.383 µs (20.9%)
iast_TELEMETRY_OFF	467.584 µs [447.097 µs, 488.071 µs]	101.482 µs (27.7%)
tracing	441.406 µs [420.238 µs, 462.575 µs]	75.304 µs (20.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	361.573 µs [341.574 µs, 381.572 µs]	-
iast	470.43 µs [449.911 µs, 490.95 µs]	108.857 µs (30.1%)
iast_FULL	533.271 µs [512.772 µs, 553.771 µs]	171.698 µs (47.5%)
iast_INACTIVE	445.973 µs [425.259 µs, 466.688 µs]	84.4 µs (23.3%)
iast_TELEMETRY_OFF	465.799 µs [445.267 µs, 486.331 µs]	104.226 µs (28.8%)
tracing	436.331 µs [416.014 µs, 456.647 µs]	74.758 µs (20.7%)

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~39a26b859c, baseline=1.28.0-SNAPSHOT~656c6600e0
dateFormat X
axisFormat %s
section baseline
no_agent (1.36 ms) : 1341, 1379
. : milestone, 1360,
appsec (1.77 ms) : 1745, 1796
. : milestone, 1770,
iast (1.531 ms) : 1506, 1555
. : milestone, 1531,
profiling (1.522 ms) : 1496, 1547
. : milestone, 1522,
tracing (1.486 ms) : 1461, 1510
. : milestone, 1486,
section candidate
no_agent (1.355 ms) : 1336, 1374
. : milestone, 1355,
appsec (1.747 ms) : 1722, 1772
. : milestone, 1747,
iast (1.494 ms) : 1470, 1519
. : milestone, 1494,
profiling (1.501 ms) : 1476, 1527
. : milestone, 1501,
tracing (1.497 ms) : 1472, 1521
. : milestone, 1497,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.36 ms [1.341 ms, 1.379 ms]	-
appsec	1.77 ms [1.745 ms, 1.796 ms]	410.197 µs (30.2%)
iast	1.531 ms [1.506 ms, 1.555 ms]	170.744 µs (12.6%)
profiling	1.522 ms [1.496 ms, 1.547 ms]	161.657 µs (11.9%)
tracing	1.486 ms [1.461 ms, 1.51 ms]	125.74 µs (9.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.355 ms [1.336 ms, 1.374 ms]	-
appsec	1.747 ms [1.722 ms, 1.772 ms]	392.381 µs (29.0%)
iast	1.494 ms [1.47 ms, 1.519 ms]	139.506 µs (10.3%)
profiling	1.501 ms [1.476 ms, 1.527 ms]	146.128 µs (10.8%)
tracing	1.497 ms [1.472 ms, 1.521 ms]	141.935 µs (10.5%)

bantonsson added tag: no release notes Changes to exclude from release notes run-tests: all Run all tests labels Jan 12, 2024

bantonsson changed the title ~~Don't create a process span for agent intternal threads~~Don't create a process span for agent internal threadsJan 12, 2024

Don't create a process span for agent internal threads
39a26b8

bantonsson force-pushed the ban/no-process-spans-for-agent branch from ebb280c to 39a26b8Compare January 12, 2024 12:55

bantonsson marked this pull request as ready for review January 12, 2024 13:48

bantonsson requested a review from a team as a code owner January 12, 2024 13:48

bantonsson requested review from PerfectSlayer and dougqh January 12, 2024 13:48

richardstartin approved these changes Jan 12, 2024
View reviewed changes

smola approved these changes Jan 12, 2024
View reviewed changes

bantonsson enabled auto-merge January 12, 2024 13:51

bantonsson disabled auto-merge January 12, 2024 15:16

bantonsson merged commit f471f93 into masterJan 12, 2024

bantonsson deleted the ban/no-process-spans-for-agent branch January 12, 2024 15:16

github-actionsBot added this to the 1.28.0 milestone Jan 12, 2024

bantonsson added inst: others All other instrumentations and removed tag: no release notes Changes to exclude from release notes labels Jan 12, 2024

smola mentioned this pull request Jan 30, 2024
Mute tracing for subprocesses executed in by the tracer #6584
Merged

Navigation Menu

Uh oh!

Don't create a process span for agent internal threads - #6478

Don't create a process span for agent internal threads#6478
bantonsson merged 1 commit into
masterfrom
ban/no-process-spans-for-agent

bantonsson commented Jan 12, 2024•
edited
Loading

Uh oh!

pr-commenterBot commented Jan 12, 2024•
edited
Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

bantonsson commented Jan 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Uh oh!

pr-commenterBot commented Jan 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bantonsson commented Jan 12, 2024•
edited
Loading

pr-commenterBot commented Jan 12, 2024•
edited
Loading