Mute tracing for subprocesses executed in by the tracer by smola · Pull Request #6584 · DataDog/dd-trace-java

smola · 2024-01-30T07:30:05Z

What Does This Do

The tracer sometimes runs external processes. Currently, hostname and getenforce. The subprocess instrumentation creates spans for these, leading to visible command_execution spans for hostname and/or enforce which may be confusing to users.

This commit uses the new blackhole span API (#6326) to mute tracing during when hostname and getenforce are run, preventing the creation of these spans.

Also reverts #6478, which would be redundant with the current cases.

Motivation

Additional Notes

Jira ticket: [PROJ-IDENT]

The tracer sometimes runs external processes in some cases. Currently, `hostname` and `getenforce`. The subprocess instrumentation creates spans for these, leading to visible `command_execution` spans for `hostname` and/or `enforce` which may be confusing to users. This commit uses the bew blackhole span API (#6326) to mute tracing during when `hostname` and `getenforce` are run, preventing the creation of these spans.

pr-commenter · 2024-01-30T08:18:59Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/mute-tracing-for-internal-process
git_commit_date	1706562398	1706599572
git_commit_sha	`281e492`	`d438c2e`
release_version	1.29.0-SNAPSHOT~281e492170	1.29.0-SNAPSHOT~d438c2e326

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1706602693	1706602693
ci_job_id	421345634	421345634
ci_pipeline_id	27381118	27381118
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 2 performance improvements and 0 performance regressions! Performance is the same for 44 metrics, 8 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:petclinic:profiling:ProfilingAgent	better [-13.928ms; -8.186ms] or [-11.095%; -6.521%]	114.475ms	125.532ms
scenario:startup:petclinic:profiling:Profiling	better [-13.928ms; -8.186ms] or [-11.093%; -6.519%]	114.500ms	125.557ms

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-01-30T07:54:20	2024-01-30T08:13:14
git_branch	master	smola/mute-tracing-for-internal-process
git_commit_date	1706562398	1706599572
git_commit_sha	`281e492`	`d438c2e`
release_version	1.29.0-SNAPSHOT~281e492170	1.29.0-SNAPSHOT~d438c2e326
start_time	2024-01-30T07:54:07	2024-01-30T08:13:01

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1706602693	1706602693
ci_job_id	421345634	421345634
ci_pipeline_id	27381118	27381118
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 16 unstable metrics.

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.29.0-SNAPSHOT~d438c2e326, baseline=1.29.0-SNAPSHOT~281e492170
dateFormat X
axisFormat %s
section baseline
no_agent (356.504 µs) : 337, 376
. : milestone, 357,
iast (478.925 µs) : 457, 500
. : milestone, 479,
iast_FULL (525.893 µs) : 505, 546
. : milestone, 526,
iast_GLOBAL (486.501 µs) : 466, 507
. : milestone, 487,
iast_HARDCODED_SECRET_DISABLED (462.414 µs) : 442, 483
. : milestone, 462,
iast_INACTIVE (438.504 µs) : 418, 459
. : milestone, 439,
iast_TELEMETRY_OFF (464.667 µs) : 444, 485
. : milestone, 465,
tracing (437.14 µs) : 416, 458
. : milestone, 437,
section candidate
no_agent (366.077 µs) : 345, 387
. : milestone, 366,
iast (472.438 µs) : 451, 493
. : milestone, 472,
iast_FULL (526.449 µs) : 506, 547
. : milestone, 526,
iast_GLOBAL (490.89 µs) : 470, 511
. : milestone, 491,
iast_HARDCODED_SECRET_DISABLED (464.475 µs) : 444, 485
. : milestone, 464,
iast_INACTIVE (443.183 µs) : 422, 464
. : milestone, 443,
iast_TELEMETRY_OFF (463.559 µs) : 443, 484
. : milestone, 464,
tracing (429.815 µs) : 410, 450
. : milestone, 430,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	356.504 µs [337.135 µs, 375.873 µs]	-
iast	478.925 µs [457.399 µs, 500.452 µs]	122.422 µs (34.3%)
iast_FULL	525.893 µs [505.358 µs, 546.428 µs]	169.39 µs (47.5%)
iast_GLOBAL	486.501 µs [465.951 µs, 507.052 µs]	129.998 µs (36.5%)
iast_HARDCODED_SECRET_DISABLED	462.414 µs [441.79 µs, 483.038 µs]	105.91 µs (29.7%)
iast_INACTIVE	438.504 µs [417.938 µs, 459.071 µs]	82.001 µs (23.0%)
iast_TELEMETRY_OFF	464.667 µs [443.946 µs, 485.389 µs]	108.164 µs (30.3%)
tracing	437.14 µs [416.444 µs, 457.837 µs]	80.637 µs (22.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.077 µs [345.103 µs, 387.051 µs]	-
iast	472.438 µs [451.396 µs, 493.481 µs]	106.361 µs (29.1%)
iast_FULL	526.449 µs [505.764 µs, 547.133 µs]	160.371 µs (43.8%)
iast_GLOBAL	490.89 µs [470.444 µs, 511.336 µs]	124.813 µs (34.1%)
iast_HARDCODED_SECRET_DISABLED	464.475 µs [444.281 µs, 484.669 µs]	98.398 µs (26.9%)
iast_INACTIVE	443.183 µs [422.221 µs, 464.145 µs]	77.106 µs (21.1%)
iast_TELEMETRY_OFF	463.559 µs [442.906 µs, 484.212 µs]	97.482 µs (26.6%)
tracing	429.815 µs [409.59 µs, 450.04 µs]	63.738 µs (17.4%)

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.29.0-SNAPSHOT~d438c2e326, baseline=1.29.0-SNAPSHOT~281e492170
dateFormat X
axisFormat %s
section baseline
no_agent (1.353 ms) : 1334, 1373
. : milestone, 1353,
appsec (1.737 ms) : 1712, 1763
. : milestone, 1737,
iast (1.519 ms) : 1495, 1544
. : milestone, 1519,
profiling (1.509 ms) : 1484, 1534
. : milestone, 1509,
tracing (1.522 ms) : 1497, 1548
. : milestone, 1522,
section candidate
no_agent (1.339 ms) : 1321, 1358
. : milestone, 1339,
appsec (1.773 ms) : 1748, 1798
. : milestone, 1773,
iast (1.497 ms) : 1473, 1522
. : milestone, 1497,
profiling (1.487 ms) : 1462, 1512
. : milestone, 1487,
tracing (1.484 ms) : 1459, 1509
. : milestone, 1484,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.353 ms [1.334 ms, 1.373 ms]	-
appsec	1.737 ms [1.712 ms, 1.763 ms]	383.88 µs (28.4%)
iast	1.519 ms [1.495 ms, 1.544 ms]	166.022 µs (12.3%)
profiling	1.509 ms [1.484 ms, 1.534 ms]	155.619 µs (11.5%)
tracing	1.522 ms [1.497 ms, 1.548 ms]	168.812 µs (12.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.339 ms [1.321 ms, 1.358 ms]	-
appsec	1.773 ms [1.748 ms, 1.798 ms]	433.509 µs (32.4%)
iast	1.497 ms [1.473 ms, 1.522 ms]	157.904 µs (11.8%)
profiling	1.487 ms [1.462 ms, 1.512 ms]	147.708 µs (11.0%)
tracing	1.484 ms [1.459 ms, 1.509 ms]	144.829 µs (10.8%)

jbachorik

Looking good for the profiler part

smola added comp: core Tracer core inst: others All other instrumentations run-tests: all Run all tests and removed comp: core Tracer core labels Jan 30, 2024

smola marked this pull request as ready for review January 30, 2024 09:18

smola requested review from a team as code owners January 30, 2024 09:18

smola requested review from PerfectSlayer and ygree January 30, 2024 09:18

jbachorik approved these changes Jan 30, 2024
View reviewed changes

mcculls approved these changes Jan 30, 2024
View reviewed changes

amarziali approved these changes Jan 30, 2024
View reviewed changes

smola merged commit 4c70643 into masterJan 30, 2024

smola deleted the smola/mute-tracing-for-internal-process branch January 30, 2024 12:07

github-actionsBot added this to the 1.29.0 milestone Jan 30, 2024

smola mentioned this pull request Feb 1, 2024
Getenforce error at JVM startup #6598
Closed

AlexeyKuznetsov-DD mentioned this pull request Jun 18, 2025
Mute tracing for subprocesses executed by bootstrap telemetry logic. #9005
Merged

Navigation Menu

Uh oh!

Mute tracing for subprocesses executed in by the tracer - #6584

Mute tracing for subprocesses executed in by the tracer#6584
smola merged 1 commit into
masterfrom
smola/mute-tracing-for-internal-process

smola commented Jan 30, 2024•
edited
Loading

Uh oh!

pr-commenterBot commented Jan 30, 2024

Uh oh!

jbachorik left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

smola commented Jan 30, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenterBot commented Jan 30, 2024

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

jbachorik left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smola commented Jan 30, 2024•
edited
Loading