Skip tests based on git changes on GitLab by smola · Pull Request #9039 · DataDog/dd-trace-java

smola · 2025-06-26T07:30:27Z

What Does This Do

Bringing back #7464 to GitLab. Tests are skipped based on Git changes. See the original PR for guidance.

Motivation

Bring down pipeline flakes, pipeline mean time, costs.

Additional Notes

Test PR: #9059

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-26T08:01:43Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/git-pr-skip
git_commit_date	1751046924	1751267928
git_commit_sha	`46d0949`	`bf4cb57`
release_version	1.51.0-SNAPSHOT~46d09492c9	1.51.0-SNAPSHOT~bf4cb576d0

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1751269751	1751269751
ci_job_id	1004190726	1004190726
ci_pipeline_id	69107919	69107919
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-tcrivaq7-project-304-concurrent-0-8lmwu9lr 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-tcrivaq7-project-304-concurrent-0-8lmwu9lr 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 44 metrics, 9 unstable metrics.

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~bf4cb576d0, baseline=1.51.0-SNAPSHOT~46d09492c9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (995.434 ms) : 0, 995434
Total [baseline] (10.653 s) : 0, 10652814
Agent [candidate] (996.384 ms) : 0, 996384
Total [candidate] (10.664 s) : 0, 10663620
section appsec
Agent [baseline] (1.181 s) : 0, 1181263
Total [baseline] (10.744 s) : 0, 10744258
Agent [candidate] (1.185 s) : 0, 1184613
Total [candidate] (10.732 s) : 0, 10731770
section iast
Agent [baseline] (1.139 s) : 0, 1138933
Total [baseline] (10.806 s) : 0, 10805610
Agent [candidate] (1.136 s) : 0, 1136320
Total [candidate] (10.804 s) : 0, 10803625
section profiling
Agent [baseline] (1.254 s) : 0, 1253998
Total [baseline] (10.915 s) : 0, 10915395
Agent [candidate] (1.261 s) : 0, 1261044
Total [candidate] (11.088 s) : 0, 11087520

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	995.434 ms	-
Agent	appsec	1.181 s	185.829 ms (18.7%)
Agent	iast	1.139 s	143.498 ms (14.4%)
Agent	profiling	1.254 s	258.564 ms (26.0%)
Total	tracing	10.653 s	-
Total	appsec	10.744 s	91.444 ms (0.9%)
Total	iast	10.806 s	152.797 ms (1.4%)
Total	profiling	10.915 s	262.582 ms (2.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	996.384 ms	-
Agent	appsec	1.185 s	188.229 ms (18.9%)
Agent	iast	1.136 s	139.936 ms (14.0%)
Agent	profiling	1.261 s	264.66 ms (26.6%)
Total	tracing	10.664 s	-
Total	appsec	10.732 s	68.149 ms (0.6%)
Total	iast	10.804 s	140.004 ms (1.3%)
Total	profiling	11.088 s	423.899 ms (4.0%)

gantt
title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~bf4cb576d0, baseline=1.51.0-SNAPSHOT~46d09492c9
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.82 ms) : 0, 686820
BytebuddyAgent [candidate] (687.69 ms) : 0, 687690
GlobalTracer [baseline] (242.488 ms) : 0, 242488
GlobalTracer [candidate] (242.515 ms) : 0, 242515
AppSec [baseline] (30.347 ms) : 0, 30347
AppSec [candidate] (30.511 ms) : 0, 30511
Debugger [baseline] (6.027 ms) : 0, 6027
Debugger [candidate] (6.025 ms) : 0, 6025
Remote Config [baseline] (675.322 µs) : 0, 675
Remote Config [candidate] (667.122 µs) : 0, 667
Telemetry [baseline] (8.239 ms) : 0, 8239
Telemetry [candidate] (8.184 ms) : 0, 8184
section appsec
BytebuddyAgent [baseline] (715.123 ms) : 0, 715123
BytebuddyAgent [candidate] (716.61 ms) : 0, 716610
GlobalTracer [baseline] (236.899 ms) : 0, 236899
GlobalTracer [candidate] (237.723 ms) : 0, 237723
IAST [baseline] (22.15 ms) : 0, 22150
IAST [candidate] (22.444 ms) : 0, 22444
AppSec [baseline] (171.615 ms) : 0, 171615
AppSec [candidate] (172.206 ms) : 0, 172206
Debugger [baseline] (5.831 ms) : 0, 5831
Debugger [candidate] (5.852 ms) : 0, 5852
Remote Config [baseline] (622.383 µs) : 0, 622
Remote Config [candidate] (628.251 µs) : 0, 628
Telemetry [baseline] (8.166 ms) : 0, 8166
Telemetry [candidate] (8.19 ms) : 0, 8190
section iast
BytebuddyAgent [baseline] (813.785 ms) : 0, 813785
BytebuddyAgent [candidate] (812.479 ms) : 0, 812479
GlobalTracer [baseline] (234.316 ms) : 0, 234316
GlobalTracer [candidate] (234.355 ms) : 0, 234355
IAST [baseline] (28.828 ms) : 0, 28828
IAST [candidate] (28.474 ms) : 0, 28474
AppSec [baseline] (26.808 ms) : 0, 26808
AppSec [candidate] (25.109 ms) : 0, 25109
Debugger [baseline] (5.841 ms) : 0, 5841
Debugger [candidate] (5.772 ms) : 0, 5772
Remote Config [baseline] (591.696 µs) : 0, 592
Remote Config [candidate] (585.682 µs) : 0, 586
Telemetry [baseline] (7.9 ms) : 0, 7900
Telemetry [candidate] (7.904 ms) : 0, 7904
section profiling
BytebuddyAgent [baseline] (684.288 ms) : 0, 684288
BytebuddyAgent [candidate] (686.242 ms) : 0, 686242
GlobalTracer [baseline] (362.812 ms) : 0, 362812
GlobalTracer [candidate] (366.181 ms) : 0, 366181
AppSec [baseline] (31.917 ms) : 0, 31917
AppSec [candidate] (32.19 ms) : 0, 32190
Debugger [baseline] (11.404 ms) : 0, 11404
Debugger [candidate] (9.928 ms) : 0, 9928
Remote Config [baseline] (1.4 ms) : 0, 1400
Remote Config [candidate] (682.299 µs) : 0, 682
Telemetry [baseline] (8.808 ms) : 0, 8808
Telemetry [candidate] (10.391 ms) : 0, 10391
ProfilingAgent [baseline] (104.348 ms) : 0, 104348
ProfilingAgent [candidate] (105.703 ms) : 0, 105703
Profiling [baseline] (104.372 ms) : 0, 104372
Profiling [candidate] (105.727 ms) : 0, 105727

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~bf4cb576d0, baseline=1.51.0-SNAPSHOT~46d09492c9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (995.151 ms) : 0, 995151
Total [baseline] (8.575 s) : 0, 8575466
Agent [candidate] (994.828 ms) : 0, 994828
Total [candidate] (8.552 s) : 0, 8551869
section iast
Agent [baseline] (1.129 s) : 0, 1129415
Total [baseline] (9.265 s) : 0, 9264645
Agent [candidate] (1.143 s) : 0, 1143325
Total [candidate] (9.354 s) : 0, 9354172

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	995.151 ms	-
Agent	iast	1.129 s	134.264 ms (13.5%)
Total	tracing	8.575 s	-
Total	iast	9.265 s	689.179 ms (8.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	994.828 ms	-
Agent	iast	1.143 s	148.497 ms (14.9%)
Total	tracing	8.552 s	-
Total	iast	9.354 s	802.303 ms (9.4%)

gantt
title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~bf4cb576d0, baseline=1.51.0-SNAPSHOT~46d09492c9
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.748 ms) : 0, 686748
BytebuddyAgent [candidate] (686.318 ms) : 0, 686318
GlobalTracer [baseline] (242.379 ms) : 0, 242379
GlobalTracer [candidate] (242.34 ms) : 0, 242340
AppSec [baseline] (30.281 ms) : 0, 30281
AppSec [candidate] (30.408 ms) : 0, 30408
Debugger [baseline] (6.016 ms) : 0, 6016
Debugger [candidate] (6.058 ms) : 0, 6058
Remote Config [baseline] (676.142 µs) : 0, 676
Remote Config [candidate] (663.201 µs) : 0, 663
Telemetry [baseline] (8.196 ms) : 0, 8196
Telemetry [candidate] (8.205 ms) : 0, 8205
section iast
BytebuddyAgent [baseline] (806.448 ms) : 0, 806448
BytebuddyAgent [candidate] (816.325 ms) : 0, 816325
GlobalTracer [baseline] (232.193 ms) : 0, 232193
GlobalTracer [candidate] (235.257 ms) : 0, 235257
AppSec [baseline] (30.346 ms) : 0, 30346
AppSec [candidate] (28.847 ms) : 0, 28847
Debugger [baseline] (6.634 ms) : 0, 6634
Debugger [candidate] (5.928 ms) : 0, 5928
Remote Config [baseline] (582.549 µs) : 0, 583
Remote Config [candidate] (595.381 µs) : 0, 595
Telemetry [baseline] (7.949 ms) : 0, 7949
Telemetry [candidate] (8.121 ms) : 0, 8121
IAST [baseline] (24.431 ms) : 0, 24431
IAST [candidate] (27.309 ms) : 0, 27309

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/git-pr-skip
git_commit_date	1751046924	1751267928
git_commit_sha	`46d0949`	`bf4cb57`
release_version	1.51.0-SNAPSHOT~46d09492c9	1.51.0-SNAPSHOT~bf4cb576d0

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1751269426	1751269426
ci_job_id	1004190727	1004190727
ci_pipeline_id	69107919	69107919
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-iscs1df1-project-304-concurrent-0-vzkiwxca 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-iscs1df1-project-304-concurrent-0-vzkiwxca 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 3 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:iast_GLOBAL:high_load	better [-687.663µs; -301.090µs] or [-6.726%; -2.945%]	unstable [-31.739op/s; +77.677op/s] or [-6.985%; +17.094%]	9.729ms	477.375op/s	10.223ms	454.406op/s
scenario:load:insecure-bank:iast:high_load	worse [+366.136µs; +705.468µs] or [+3.963%; +7.636%]	unstable [-84.405op/s; +29.842op/s] or [-16.803%; +5.941%]	9.775ms	475.031op/s	9.239ms	502.312op/s
scenario:load:petclinic:no_agent:high_load	worse [+1.160ms; +1.821ms] or [+3.162%; +4.961%]	unstable [-13.596op/s; +3.746op/s] or [-10.674%; +2.941%]	38.184ms	122.450op/s	36.694ms	127.375op/s
scenario:load:petclinic:code_origins:high_load	worse [+1.597ms; +2.413ms] or [+3.730%; +5.635%]	unstable [-12.126op/s; +2.376op/s] or [-11.103%; +2.176%]	44.829ms	104.338op/s	42.824ms	109.213op/s

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~bf4cb576d0, baseline=1.51.0-SNAPSHOT~46d09492c9
dateFormat X
axisFormat %s
section baseline
no_agent (4.266 ms) : 4217, 4314
. : milestone, 4266,
iast (9.239 ms) : 9089, 9389
. : milestone, 9239,
iast_FULL (13.666 ms) : 13398, 13934
. : milestone, 13666,
iast_GLOBAL (10.223 ms) : 10036, 10411
. : milestone, 10223,
profiling (8.687 ms) : 8554, 8819
. : milestone, 8687,
tracing (7.423 ms) : 7311, 7535
. : milestone, 7423,
section candidate
no_agent (4.337 ms) : 4288, 4385
. : milestone, 4337,
iast (9.775 ms) : 9610, 9939
. : milestone, 9775,
iast_FULL (14.218 ms) : 13933, 14503
. : milestone, 14218,
iast_GLOBAL (9.729 ms) : 9558, 9900
. : milestone, 9729,
profiling (8.611 ms) : 8479, 8744
. : milestone, 8611,
tracing (7.648 ms) : 7532, 7764
. : milestone, 7648,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.266 ms [4.217 ms, 4.314 ms]	-
iast	9.239 ms [9.089 ms, 9.389 ms]	4.973 ms (116.6%)
iast_FULL	13.666 ms [13.398 ms, 13.934 ms]	9.4 ms (220.4%)
iast_GLOBAL	10.223 ms [10.036 ms, 10.411 ms]	5.958 ms (139.7%)
profiling	8.687 ms [8.554 ms, 8.819 ms]	4.421 ms (103.7%)
tracing	7.423 ms [7.311 ms, 7.535 ms]	3.158 ms (74.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.337 ms [4.288 ms, 4.385 ms]	-
iast	9.775 ms [9.61 ms, 9.939 ms]	5.438 ms (125.4%)
iast_FULL	14.218 ms [13.933 ms, 14.503 ms]	9.881 ms (227.9%)
iast_GLOBAL	9.729 ms [9.558 ms, 9.9 ms]	5.393 ms (124.3%)
profiling	8.611 ms [8.479 ms, 8.744 ms]	4.275 ms (98.6%)
tracing	7.648 ms [7.532 ms, 7.764 ms]	3.311 ms (76.4%)

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~bf4cb576d0, baseline=1.51.0-SNAPSHOT~46d09492c9
dateFormat X
axisFormat %s
section baseline
no_agent (36.694 ms) : 36390, 36997
. : milestone, 36694,
appsec (46.377 ms) : 45977, 46777
. : milestone, 46377,
code_origins (42.824 ms) : 42466, 43181
. : milestone, 42824,
iast (46.031 ms) : 45615, 46447
. : milestone, 46031,
profiling (48.354 ms) : 47931, 48777
. : milestone, 48354,
tracing (41.964 ms) : 41611, 42316
. : milestone, 41964,
section candidate
no_agent (38.184 ms) : 37874, 38494
. : milestone, 38184,
appsec (46.61 ms) : 46204, 47016
. : milestone, 46610,
code_origins (44.829 ms) : 44430, 45228
. : milestone, 44829,
iast (46.298 ms) : 45882, 46713
. : milestone, 46298,
profiling (47.077 ms) : 46643, 47510
. : milestone, 47077,
tracing (41.938 ms) : 41586, 42289
. : milestone, 41938,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.694 ms [36.39 ms, 36.997 ms]	-
appsec	46.377 ms [45.977 ms, 46.777 ms]	9.683 ms (26.4%)
code_origins	42.824 ms [42.466 ms, 43.181 ms]	6.13 ms (16.7%)
iast	46.031 ms [45.615 ms, 46.447 ms]	9.338 ms (25.4%)
profiling	48.354 ms [47.931 ms, 48.777 ms]	11.66 ms (31.8%)
tracing	41.964 ms [41.611 ms, 42.316 ms]	5.27 ms (14.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	38.184 ms [37.874 ms, 38.494 ms]	-
appsec	46.61 ms [46.204 ms, 47.016 ms]	8.426 ms (22.1%)
code_origins	44.829 ms [44.43 ms, 45.228 ms]	6.645 ms (17.4%)
iast	46.298 ms [45.882 ms, 46.713 ms]	8.114 ms (21.2%)
profiling	47.077 ms [46.643 ms, 47.51 ms]	8.893 ms (23.3%)
tracing	41.938 ms [41.586 ms, 42.289 ms]	3.754 ms (9.8%)

Dacapo

bric3 · 2025-06-30T10:14:23Z

question: Did you thought about making this part of our gradle itself. Not that I'm asking for it specifically. As I would like to avoid adding yet another groovy scripting at this time.
Also, I wonder does our gitlab CI have this resolved ?
cc @randomanderson

Did you thought about making this part of our gradle itself.
Can be done at gradle level. A bash script for CI seemed pretty self-contained and easily debuggable though.
Also, I wonder does our gitlab CI have this resolved ?
Not supported by GitLab, because GitHub PRs are not native to GitLab CI, and we do not use GitLab MR. We have an internal solution at DD that requires migrating to a new CI system for GitLab. That's apparently a lot of work for the repo since it was not used since the beginning.
In our current setup, this is the recommended way. I took it from dd-trace-py.

Ah thanks for the info !

smola added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jun 26, 2025

smola force-pushed the smola/git-pr-skip branch from 31211b9 to 04a06f8Compare June 26, 2025 08:26

smola added 2 commits June 27, 2025 17:48

Skip tests based on git changes on GitLab
3fc64a1

Use DDCI_PULL_REQUEST_TARGET_SHA
283207e

smola force-pushed the smola/git-pr-skip branch from 04a06f8 to 283207eCompare June 27, 2025 15:51

smola added 4 commits June 27, 2025 19:11

Add script to infer base reference
15a1ad8

Infer base branch
a1ec019

Fetch GitHub token from AWS SSM
0dccb3d

Fix AWS SSM token name
b06a47b

smola marked this pull request as ready for review June 30, 2025 06:54

smola requested a review from a team as a code owner June 30, 2025 06:54

smola requested review from TonyCTHsu and removed request for a teamJune 30, 2025 06:54

bric3 reviewed Jun 30, 2025
View reviewed changes

Comment thread.gitlab-ci.yml Outdated

Extract gitlab_base_ref_params
bf4cb57

smola requested a review from bric3 June 30, 2025 07:19

bric3 reviewed Jun 30, 2025
View reviewed changes

bric3 approved these changes Jun 30, 2025
View reviewed changes

smola merged commit 7494696 into masterJun 30, 2025

smola deleted the smola/git-pr-skip branch June 30, 2025 11:58

github-actionsBot added this to the 1.51.0 milestone Jun 30, 2025

PerfectSlayer added the type: feature Enhancements and improvements label Jul 1, 2025

smola mentioned this pull request Jul 1, 2025
Disable skipping tests based on git changes #9078
Merged

smola added a commit that referenced this pull request Jul 2, 2025
Skip tests based on git changes on GitLab (#9039)
6a78f3a

smola mentioned this pull request Jul 2, 2025
Find PR base branch without GitHub API and enable test skipping #9080
Merged

Navigation Menu

Uh oh!

Skip tests based on git changes on GitLab - #9039

Skip tests based on git changes on GitLab#9039
smola merged 7 commits into
masterfrom
smola/git-pr-skip

smola commented Jun 26, 2025•
edited
Loading

Uh oh!

pr-commenterBot commented Jun 26, 2025•
edited
Loading

Uh oh!

Uh oh!

bric3Jun 30, 2025

Uh oh!

smolaJun 30, 2025•
edited
Loading

Uh oh!

bric3Jun 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

smola commented Jun 26, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenterBot commented Jun 26, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Uh oh!

Uh oh!

bric3Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

smolaJun 30, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bric3Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smola commented Jun 26, 2025•
edited
Loading

pr-commenterBot commented Jun 26, 2025•
edited
Loading

smolaJun 30, 2025•
edited
Loading