Implement test disabling by daniel-mohedano · Pull Request #8377 · DataDog/dd-trace-java

daniel-mohedano · 2025-02-12T15:56:43Z

What Does This Do

Implements test disabling as part of Flaky Test Management. A disabled test will simply be skipped.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1625

pr-commenter · 2025-02-12T16:34:09Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/disabled-tests
git_commit_date	1739448559	1739448583
git_commit_sha	`c9b57a0`	`b98721e`
release_version	1.47.0-SNAPSHOT~c9b57a0ca5	1.47.0-SNAPSHOT~b98721e1e8

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1739451038	1739451038
ci_job_id	806821251	806821251
ci_pipeline_id	55802563	55802563
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-f42mxumf-project-304-concurrent-0-ori4p657 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-f42mxumf-project-304-concurrent-0-ori4p657 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 4 unstable metrics.

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.039 s) : 0, 1039166
Total [baseline] (10.407 s) : 0, 10406751
Agent [candidate] (1.052 s) : 0, 1051702
Total [candidate] (10.544 s) : 0, 10543959
section appsec
Agent [baseline] (1.183 s) : 0, 1183161
Total [baseline] (10.72 s) : 0, 10719840
Agent [candidate] (1.183 s) : 0, 1183147
Total [candidate] (10.782 s) : 0, 10782307
section iast
Agent [baseline] (1.169 s) : 0, 1169140
Total [baseline] (11.005 s) : 0, 11005358
Agent [candidate] (1.172 s) : 0, 1172479
Total [candidate] (10.988 s) : 0, 10987844
section profiling
Agent [baseline] (1.262 s) : 0, 1262304
Total [baseline] (10.855 s) : 0, 10854966
Agent [candidate] (1.259 s) : 0, 1258961
Total [candidate] (10.816 s) : 0, 10816364

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.039 s	-
Agent	appsec	1.183 s	143.995 ms (13.9%)
Agent	iast	1.169 s	129.973 ms (12.5%)
Agent	profiling	1.262 s	223.138 ms (21.5%)
Total	tracing	10.407 s	-
Total	appsec	10.72 s	313.088 ms (3.0%)
Total	iast	11.005 s	598.607 ms (5.8%)
Total	profiling	10.855 s	448.215 ms (4.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.052 s	-
Agent	appsec	1.183 s	131.445 ms (12.5%)
Agent	iast	1.172 s	120.777 ms (11.5%)
Agent	profiling	1.259 s	207.259 ms (19.7%)
Total	tracing	10.544 s	-
Total	appsec	10.782 s	238.347 ms (2.3%)
Total	iast	10.988 s	443.884 ms (4.2%)
Total	profiling	10.816 s	272.404 ms (2.6%)

gantt
title petclinic - break down per module: candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (715.922 ms) : 0, 715922
BytebuddyAgent [candidate] (723.803 ms) : 0, 723803
GlobalTracer [baseline] (240.191 ms) : 0, 240191
GlobalTracer [candidate] (242.932 ms) : 0, 242932
AppSec [baseline] (55.7 ms) : 0, 55700
AppSec [candidate] (55.861 ms) : 0, 55861
Remote Config [baseline] (721.706 µs) : 0, 722
Remote Config [candidate] (727.689 µs) : 0, 728
Telemetry [baseline] (11.392 ms) : 0, 11392
Telemetry [candidate] (13.058 ms) : 0, 13058
section appsec
BytebuddyAgent [baseline] (734.568 ms) : 0, 734568
BytebuddyAgent [candidate] (734.07 ms) : 0, 734070
GlobalTracer [baseline] (237.62 ms) : 0, 237620
GlobalTracer [candidate] (237.543 ms) : 0, 237543
AppSec [baseline] (175.965 ms) : 0, 175965
AppSec [candidate] (176.508 ms) : 0, 176508
Remote Config [baseline] (645.094 µs) : 0, 645
Remote Config [candidate] (653.002 µs) : 0, 653
Telemetry [baseline] (8.269 ms) : 0, 8269
Telemetry [candidate] (8.256 ms) : 0, 8256
IAST [baseline] (21.649 ms) : 0, 21649
IAST [candidate] (21.793 ms) : 0, 21793
section iast
BytebuddyAgent [baseline] (834.355 ms) : 0, 834355
BytebuddyAgent [candidate] (836.514 ms) : 0, 836514
GlobalTracer [baseline] (230.472 ms) : 0, 230472
GlobalTracer [candidate] (231.36 ms) : 0, 231360
AppSec [baseline] (56.836 ms) : 0, 56836
AppSec [candidate] (56.965 ms) : 0, 56965
Remote Config [baseline] (604.28 µs) : 0, 604
Remote Config [candidate] (609.829 µs) : 0, 610
Telemetry [baseline] (8.723 ms) : 0, 8723
Telemetry [candidate] (8.771 ms) : 0, 8771
IAST [baseline] (22.893 ms) : 0, 22893
IAST [candidate] (22.951 ms) : 0, 22951
section profiling
BytebuddyAgent [baseline] (708.399 ms) : 0, 708399
BytebuddyAgent [candidate] (706.249 ms) : 0, 706249
GlobalTracer [baseline] (350.308 ms) : 0, 350308
GlobalTracer [candidate] (350.906 ms) : 0, 350906
AppSec [baseline] (55.422 ms) : 0, 55422
AppSec [candidate] (54.632 ms) : 0, 54632
Remote Config [baseline] (699.817 µs) : 0, 700
Remote Config [candidate] (704.369 µs) : 0, 704
Telemetry [baseline] (8.923 ms) : 0, 8923
Telemetry [candidate] (8.859 ms) : 0, 8859
ProfilingAgent [baseline] (96.189 ms) : 0, 96189
ProfilingAgent [candidate] (95.386 ms) : 0, 95386
Profiling [baseline] (96.213 ms) : 0, 96213
Profiling [candidate] (95.412 ms) : 0, 95412

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1043041
Total [baseline] (8.652 s) : 0, 8651646
Agent [candidate] (1.036 s) : 0, 1036043
Total [candidate] (8.625 s) : 0, 8625186
section iast
Agent [baseline] (1.171 s) : 0, 1171331
Total [baseline] (9.269 s) : 0, 9268598
Agent [candidate] (1.17 s) : 0, 1170484
Total [candidate] (9.239 s) : 0, 9239087
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.17 s) : 0, 1170205
Total [baseline] (9.194 s) : 0, 9193977
Agent [candidate] (1.182 s) : 0, 1182268
Total [candidate] (9.299 s) : 0, 9298849
section iast_TELEMETRY_OFF
Agent [baseline] (1.173 s) : 0, 1173039
Total [baseline] (9.266 s) : 0, 9265753
Agent [candidate] (1.169 s) : 0, 1168633
Total [candidate] (9.225 s) : 0, 9225241

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	iast	1.171 s	128.29 ms (12.3%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.17 s	127.164 ms (12.2%)
Agent	iast_TELEMETRY_OFF	1.173 s	129.998 ms (12.5%)
Total	tracing	8.652 s	-
Total	iast	9.269 s	616.952 ms (7.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.194 s	542.331 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.266 s	614.107 ms (7.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.036 s	-
Agent	iast	1.17 s	134.44 ms (13.0%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.182 s	146.225 ms (14.1%)
Agent	iast_TELEMETRY_OFF	1.169 s	132.59 ms (12.8%)
Total	tracing	8.625 s	-
Total	iast	9.239 s	613.902 ms (7.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.299 s	673.664 ms (7.8%)
Total	iast_TELEMETRY_OFF	9.225 s	600.055 ms (7.0%)

gantt
title insecure-bank - break down per module: candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (719.023 ms) : 0, 719023
BytebuddyAgent [candidate] (714.704 ms) : 0, 714704
GlobalTracer [baseline] (240.943 ms) : 0, 240943
GlobalTracer [candidate] (240.186 ms) : 0, 240186
AppSec [baseline] (55.697 ms) : 0, 55697
AppSec [candidate] (55.352 ms) : 0, 55352
Remote Config [baseline] (718.701 µs) : 0, 719
Remote Config [candidate] (716.812 µs) : 0, 717
Telemetry [baseline] (11.482 ms) : 0, 11482
Telemetry [candidate] (9.948 ms) : 0, 9948
section iast
BytebuddyAgent [baseline] (835.317 ms) : 0, 835317
BytebuddyAgent [candidate] (834.946 ms) : 0, 834946
GlobalTracer [baseline] (231.261 ms) : 0, 231261
GlobalTracer [candidate] (231.183 ms) : 0, 231183
IAST [baseline] (22.919 ms) : 0, 22919
IAST [candidate] (22.904 ms) : 0, 22904
AppSec [baseline] (57.341 ms) : 0, 57341
AppSec [candidate] (56.942 ms) : 0, 56942
Remote Config [baseline] (610.008 µs) : 0, 610
Remote Config [candidate] (608.031 µs) : 0, 608
Telemetry [baseline] (8.74 ms) : 0, 8740
Telemetry [candidate] (8.683 ms) : 0, 8683
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (834.35 ms) : 0, 834350
BytebuddyAgent [candidate] (842.298 ms) : 0, 842298
GlobalTracer [baseline] (231.035 ms) : 0, 231035
GlobalTracer [candidate] (233.964 ms) : 0, 233964
IAST [baseline] (22.963 ms) : 0, 22963
IAST [candidate] (23.247 ms) : 0, 23247
AppSec [baseline] (57.246 ms) : 0, 57246
AppSec [candidate] (57.806 ms) : 0, 57806
Remote Config [baseline] (617.678 µs) : 0, 618
Remote Config [candidate] (627.598 µs) : 0, 628
Telemetry [baseline] (8.682 ms) : 0, 8682
Telemetry [candidate] (8.896 ms) : 0, 8896
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (836.799 ms) : 0, 836799
BytebuddyAgent [candidate] (833.66 ms) : 0, 833660
GlobalTracer [baseline] (232.096 ms) : 0, 232096
GlobalTracer [candidate] (231.256 ms) : 0, 231256
IAST [baseline] (25.502 ms) : 0, 25502
IAST [candidate] (26.199 ms) : 0, 26199
AppSec [baseline] (54.077 ms) : 0, 54077
AppSec [candidate] (53.047 ms) : 0, 53047
Remote Config [baseline] (617.075 µs) : 0, 617
Remote Config [candidate] (608.217 µs) : 0, 608
Telemetry [baseline] (8.698 ms) : 0, 8698
Telemetry [candidate] (8.61 ms) : 0, 8610

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-02-13T12:20:54	2025-02-13T12:27:58
git_branch	master	daniel.mohedano/disabled-tests
git_commit_date	1739448559	1739448583
git_commit_sha	`c9b57a0`	`b98721e`
release_version	1.47.0-SNAPSHOT~c9b57a0ca5	1.47.0-SNAPSHOT~b98721e1e8
start_time	2025-02-13T12:20:40	2025-02-13T12:27:44

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1739450036	1739450036
ci_job_id	806821252	806821252
ci_pipeline_id	55802563	55802563
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-f42mxumf-project-304-concurrent-1-u13y3kp9 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-f42mxumf-project-304-concurrent-1-u13y3kp9 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section baseline
no_agent (373.314 µs) : 354, 393
. : milestone, 373,
iast (507.159 µs) : 485, 529
. : milestone, 507,
iast_FULL (743.724 µs) : 722, 766
. : milestone, 744,
iast_GLOBAL (563.203 µs) : 541, 585
. : milestone, 563,
iast_HARDCODED_SECRET_DISABLED (505.914 µs) : 484, 528
. : milestone, 506,
iast_INACTIVE (452.478 µs) : 431, 474
. : milestone, 452,
iast_TELEMETRY_OFF (496.811 µs) : 475, 518
. : milestone, 497,
tracing (451.988 µs) : 431, 473
. : milestone, 452,
section candidate
no_agent (375.234 µs) : 355, 395
. : milestone, 375,
iast (513.829 µs) : 492, 536
. : milestone, 514,
iast_FULL (744.956 µs) : 723, 767
. : milestone, 745,
iast_GLOBAL (557.405 µs) : 534, 580
. : milestone, 557,
iast_HARDCODED_SECRET_DISABLED (517.887 µs) : 494, 542
. : milestone, 518,
iast_INACTIVE (451.168 µs) : 430, 472
. : milestone, 451,
iast_TELEMETRY_OFF (498.612 µs) : 475, 522
. : milestone, 499,
tracing (451.504 µs) : 430, 473
. : milestone, 452,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	373.314 µs [353.61 µs, 393.018 µs]	-
iast	507.159 µs [485.421 µs, 528.898 µs]	133.846 µs (35.9%)
iast_FULL	743.724 µs [721.895 µs, 765.553 µs]	370.41 µs (99.2%)
iast_GLOBAL	563.203 µs [541.225 µs, 585.182 µs]	189.89 µs (50.9%)
iast_HARDCODED_SECRET_DISABLED	505.914 µs [484.168 µs, 527.66 µs]	132.6 µs (35.5%)
iast_INACTIVE	452.478 µs [431.392 µs, 473.565 µs]	79.165 µs (21.2%)
iast_TELEMETRY_OFF	496.811 µs [475.138 µs, 518.483 µs]	123.497 µs (33.1%)
tracing	451.988 µs [431.235 µs, 472.741 µs]	78.674 µs (21.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	375.234 µs [355.122 µs, 395.346 µs]	-
iast	513.829 µs [491.867 µs, 535.791 µs]	138.595 µs (36.9%)
iast_FULL	744.956 µs [722.984 µs, 766.929 µs]	369.722 µs (98.5%)
iast_GLOBAL	557.405 µs [534.495 µs, 580.316 µs]	182.171 µs (48.5%)
iast_HARDCODED_SECRET_DISABLED	517.887 µs [493.531 µs, 542.244 µs]	142.653 µs (38.0%)
iast_INACTIVE	451.168 µs [430.292 µs, 472.044 µs]	75.934 µs (20.2%)
iast_TELEMETRY_OFF	498.612 µs [475.165 µs, 522.058 µs]	123.378 µs (32.9%)
tracing	451.504 µs [429.817 µs, 473.19 µs]	76.27 µs (20.3%)

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section baseline
no_agent (1.36 ms) : 1341, 1379
. : milestone, 1360,
appsec (1.744 ms) : 1720, 1767
. : milestone, 1744,
appsec_no_iast (1.74 ms) : 1715, 1765
. : milestone, 1740,
iast (1.502 ms) : 1478, 1526
. : milestone, 1502,
profiling (1.505 ms) : 1482, 1529
. : milestone, 1505,
tracing (1.488 ms) : 1463, 1513
. : milestone, 1488,
section candidate
no_agent (1.356 ms) : 1336, 1377
. : milestone, 1356,
appsec (1.754 ms) : 1731, 1777
. : milestone, 1754,
appsec_no_iast (1.75 ms) : 1725, 1775
. : milestone, 1750,
iast (1.507 ms) : 1483, 1530
. : milestone, 1507,
profiling (1.508 ms) : 1484, 1532
. : milestone, 1508,
tracing (1.491 ms) : 1467, 1515
. : milestone, 1491,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.36 ms [1.341 ms, 1.379 ms]	-
appsec	1.744 ms [1.72 ms, 1.767 ms]	383.77 µs (28.2%)
appsec_no_iast	1.74 ms [1.715 ms, 1.765 ms]	380.1 µs (27.9%)
iast	1.502 ms [1.478 ms, 1.526 ms]	142.026 µs (10.4%)
profiling	1.505 ms [1.482 ms, 1.529 ms]	145.199 µs (10.7%)
tracing	1.488 ms [1.463 ms, 1.513 ms]	127.897 µs (9.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.356 ms [1.336 ms, 1.377 ms]	-
appsec	1.754 ms [1.731 ms, 1.777 ms]	397.887 µs (29.3%)
appsec_no_iast	1.75 ms [1.725 ms, 1.775 ms]	393.77 µs (29.0%)
iast	1.507 ms [1.483 ms, 1.53 ms]	150.405 µs (11.1%)
profiling	1.508 ms [1.484 ms, 1.532 ms]	151.941 µs (11.2%)
tracing	1.491 ms [1.467 ms, 1.515 ms]	134.932 µs (9.9%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/disabled-tests
git_commit_date	1739448559	1739448583
git_commit_sha	`c9b57a0`	`b98721e`
release_version	1.47.0-SNAPSHOT~c9b57a0ca5	1.47.0-SNAPSHOT~b98721e1e8

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1739450619	1739450619
ci_job_id	806821253	806821253
ci_pipeline_id	55802563	55802563
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-gkaizwfk-project-304-concurrent-0-iwdv5dkr 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-gkaizwfk-project-304-concurrent-0-iwdv5dkr 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section baseline
no_agent (15.393 s) : 15393000, 15393000
. : milestone, 15393000,
appsec (15.014 s) : 15014000, 15014000
. : milestone, 15014000,
iast (18.503 s) : 18503000, 18503000
. : milestone, 18503000,
iast_GLOBAL (17.692 s) : 17692000, 17692000
. : milestone, 17692000,
profiling (15.19 s) : 15190000, 15190000
. : milestone, 15190000,
tracing (14.858 s) : 14858000, 14858000
. : milestone, 14858000,
section candidate
no_agent (15.271 s) : 15271000, 15271000
. : milestone, 15271000,
appsec (15.125 s) : 15125000, 15125000
. : milestone, 15125000,
iast (18.631 s) : 18631000, 18631000
. : milestone, 18631000,
iast_GLOBAL (17.923 s) : 17923000, 17923000
. : milestone, 17923000,
profiling (15.179 s) : 15179000, 15179000
. : milestone, 15179000,
tracing (15.01 s) : 15010000, 15010000
. : milestone, 15010000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.393 s [15.393 s, 15.393 s]	-
appsec	15.014 s [15.014 s, 15.014 s]	-379.0 ms (-2.5%)
iast	18.503 s [18.503 s, 18.503 s]	3.11 s (20.2%)
iast_GLOBAL	17.692 s [17.692 s, 17.692 s]	2.299 s (14.9%)
profiling	15.19 s [15.19 s, 15.19 s]	-203.0 ms (-1.3%)
tracing	14.858 s [14.858 s, 14.858 s]	-535.0 ms (-3.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.271 s [15.271 s, 15.271 s]	-
appsec	15.125 s [15.125 s, 15.125 s]	-146.0 ms (-1.0%)
iast	18.631 s [18.631 s, 18.631 s]	3.36 s (22.0%)
iast_GLOBAL	17.923 s [17.923 s, 17.923 s]	2.652 s (17.4%)
profiling	15.179 s [15.179 s, 15.179 s]	-92.0 ms (-0.6%)
tracing	15.01 s [15.01 s, 15.01 s]	-261.0 ms (-1.7%)

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~b98721e1e8, baseline=1.47.0-SNAPSHOT~c9b57a0ca5
dateFormat X
axisFormat %s
section baseline
no_agent (1.467 ms) : 1456, 1479
. : milestone, 1467,
appsec (2.345 ms) : 2302, 2388
. : milestone, 2345,
iast (2.101 ms) : 2046, 2156
. : milestone, 2101,
iast_GLOBAL (2.145 ms) : 2090, 2200
. : milestone, 2145,
profiling (1.949 ms) : 1906, 1992
. : milestone, 1949,
tracing (1.946 ms) : 1904, 1988
. : milestone, 1946,
section candidate
no_agent (1.467 ms) : 1456, 1478
. : milestone, 1467,
appsec (2.352 ms) : 2308, 2395
. : milestone, 2352,
iast (2.105 ms) : 2050, 2159
. : milestone, 2105,
iast_GLOBAL (2.153 ms) : 2097, 2208
. : milestone, 2153,
profiling (1.964 ms) : 1920, 2008
. : milestone, 1964,
tracing (1.939 ms) : 1897, 1981
. : milestone, 1939,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.467 ms [1.456 ms, 1.479 ms]	-
appsec	2.345 ms [2.302 ms, 2.388 ms]	877.683 µs (59.8%)
iast	2.101 ms [2.046 ms, 2.156 ms]	633.651 µs (43.2%)
iast_GLOBAL	2.145 ms [2.09 ms, 2.2 ms]	677.207 µs (46.1%)
profiling	1.949 ms [1.906 ms, 1.992 ms]	481.873 µs (32.8%)
tracing	1.946 ms [1.904 ms, 1.988 ms]	478.591 µs (32.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.467 ms [1.456 ms, 1.478 ms]	-
appsec	2.352 ms [2.308 ms, 2.395 ms]	884.797 µs (60.3%)
iast	2.105 ms [2.05 ms, 2.159 ms]	637.671 µs (43.5%)
iast_GLOBAL	2.153 ms [2.097 ms, 2.208 ms]	685.719 µs (46.7%)
profiling	1.964 ms [1.92 ms, 2.008 ms]	496.852 µs (33.9%)
tracing	1.939 ms [1.897 ms, 1.981 ms]	471.997 µs (32.2%)

nikita-tkachenko-datadog

LGTM, other than a couple of tiny nitpicks

nikita-tkachenko-datadog · 2025-02-12T16:29:14Z

 }

+ def "test disabled #testcaseName"() {
+ Assumptions.assumeTrue(isExceptionSuppressionSupported())


We don't use exception suppression mechanism for skipping tests (rather, it is used for running a test and ignoring possible failures), so this assumption can be removed

nikita-tkachenko-datadog · 2025-02-12T16:31:20Z

 where:
 projectName | mavenVersion
- "test_successful_maven_run_quarantined_tests" | "3.9.9"
+ "test_successful_maven_run_test_management" | "3.9.9"


Nitpick: table formatting

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | --- ### Release Notes <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.47.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.47.0): 1.47.0 ##### Components ##### Application Security Management (IAST) - 🐛 Exclude com.stripe.net.HttpURLConnectionClient to solve IAST SSRF vulnerability false positives ([#8483](DataDog/dd-trace-java#8483) - [@jandro996](https://github.com/jandro996)) - 🐛 Add exclusion to solve IAST weak randomness vulnerability false positives ([#8462](DataDog/dd-trace-java#8462) - [@jandro996](https://github.com/jandro996)) - ✨ Fix weak randomness false positive in Kafka client ([#8408](DataDog/dd-trace-java#8408) - [@smola](https://github.com/smola)) - ✨ Fix location for SSRF with Kong Unirest ([#8407](DataDog/dd-trace-java#8407) - [@smola](https://github.com/smola)) - ✨ Exclude IBM Instana from IAST ([#8406](DataDog/dd-trace-java#8406) - [@smola](https://github.com/smola)) - 🐛 Fix org.json iast instrumentation test for latest dependency ([#8347](DataDog/dd-trace-java#8347) - [@jandro996](https://github.com/jandro996)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) - ✨ Address cookie vulnerability cardinality issues ([#8210](DataDog/dd-trace-java#8210) - [@jandro996](https://github.com/jandro996)) - ✨ Email HTML Injection detection in IAST ([#8205](DataDog/dd-trace-java#8205) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Application Security Management (WAF) - 🐛✨ Ensure usr.exists tag is not overridden when UsernameNotFoundException is thrown ([#8376](DataDog/dd-trace-java#8376) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛✨ Ensure usr.exists tag is not overridden by auto instrumentation ([#8374](DataDog/dd-trace-java#8374) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update appsec metrics with event_rules_version tag ([#8354](DataDog/dd-trace-java#8354) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) - ✨ Improve ASM support in vert.x 5.0 ([#8285](DataDog/dd-trace-java#8285) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update metrics: appsec.waf.updates and appsec.waf.init ([#8280](DataDog/dd-trace-java#8280) - [@Mariovido](https://github.com/Mariovido)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) ##### Build & Tooling - 🐛 Do not generate Muzzle references for primitive arrays in method body ([#8361](DataDog/dd-trace-java#8361) - [@amarziali](https://github.com/amarziali)) - 📖 Improve dev env setup documentation for Windows ([#8180](DataDog/dd-trace-java#8180) - [@lucaspimentel](https://github.com/lucaspimentel)) ##### Continuous Integration Visibility - ✨ Add support for skip-EFD tagging ([#8487](DataDog/dd-trace-java#8487) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix an NPE in Gradle Android instrumentation ([#8484](DataDog/dd-trace-java#8484) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Consider modified tests when applying fail-fast tests ordering ([#8474](DataDog/dd-trace-java#8474) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests reordering for TestNG ([#8467](DataDog/dd-trace-java#8467) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Gradle Launcher instrumentation to not interfere with Gradle Test Kit ([#8465](DataDog/dd-trace-java#8465) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Use separate TestEventHandlers per framework in CI Vis instrumentations ([#8451](DataDog/dd-trace-java#8451) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Remove warning log when JUnit 4 test method cannot be retrieved ([#8445](DataDog/dd-trace-java#8445) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Scalatest tracing for tests that are reported asynchronously ([#8444](DataDog/dd-trace-java#8444) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement attempt to fix tests ([#8393](DataDog/dd-trace-java#8393) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement test disabling ([#8377](DataDog/dd-trace-java#8377) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Update CODEOWNERS parser to not log errors on comments with leading whitespace ([#8349](DataDog/dd-trace-java#8349) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Request Test Management tests list ([#8345](DataDog/dd-trace-java#8345) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Receive test management settings from CIVis settings request ([#8331](DataDog/dd-trace-java#8331) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement quarantined tests tagging ([#8326](DataDog/dd-trace-java#8326) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests quarantining ([#8320](DataDog/dd-trace-java#8320) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add tag to specify if the user is setting DD_SERVICE ([#8318](DataDog/dd-trace-java#8318) - [@daniel-mohedano](https://github.com/daniel-mohedano)) ##### Crash tracking - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Use Java home of the crashed process to launch crash uploader ([#8348](DataDog/dd-trace-java#8348) - [@jbachorik](https://github.com/jbachorik)) ##### Data Streams Monitoring - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix service name overrides in consumers ([#8387](DataDog/dd-trace-java#8387) - [@piochelepiotr](https://github.com/piochelepiotr)) ##### Database Monitoring - ✨ Add DBMTracePreparedStatements to tracer configuration log ([#8508](DataDog/dd-trace-java#8508) - [@cecile75](https://github.com/cecile75)) ##### Dynamic Instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Fix Exception Replay with Lambda proxy classes ([#8452](DataDog/dd-trace-java#8452) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add support for scanning jar from loaded class ([#8370](DataDog/dd-trace-java#8370) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Disable capture of entry values ([#8369](DataDog/dd-trace-java#8369) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix CodeOrigin for `@Trace` annotation ([#8344](DataDog/dd-trace-java#8344) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix equals/hashCode for CodeOrigin probe ([#8319](DataDog/dd-trace-java#8319) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### Metrics - ✨ Create metric: appsec.waf.error ([#8381](DataDog/dd-trace-java#8381) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Create metric: appsec.rasp.error ([#8364](DataDog/dd-trace-java#8364) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Profiling - ✨ Bump ddprof library to 1.22.0 ([#8463](DataDog/dd-trace-java#8463) - [@jbachorik](https://github.com/jbachorik)) - IBM J9 8u361 corresponds to OpenJDK 8u362 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#187 - Fix compatibility with musl libc 1.2.4 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#189 - Modify version extraction by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#179 - Do not write null values to jvminfo event by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#184 - Productize VMStructs-based stack walker by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#177 - A few minor downport issues by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#180 - Enable ASGCT by default on fairly safe J9 JDK versions by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#181 - 🐛 Exclude OrderedThreadPoolExecutor from queue-time measurements ([#8456](DataDog/dd-trace-java#8456) - [@jbachorik](https://github.com/jbachorik)) - ✨ Record JVM info on JVMs without JFR ([#8431](DataDog/dd-trace-java#8431) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Actually use CleanupTask in TempLocationManager ([#8420](DataDog/dd-trace-java#8420) - [@mcculls](https://github.com/mcculls)) - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Adjust JFR checks for J9 ([#8405](DataDog/dd-trace-java#8405) - [@jbachorik](https://github.com/jbachorik)) - 🧹 Disable smap RSS parsing by default ([#8342](DataDog/dd-trace-java#8342) - [@MattAlp](https://github.com/MattAlp)) ##### Telemetry - 🐛 Add support for JBoss jar:file format to DependencyResolver ([#8428](DataDog/dd-trace-java#8428) - [@jandro996](https://github.com/jandro996)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) ##### Trace context propagation - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - 🐛 Fix Stable Config telemetry source names ([#8460](DataDog/dd-trace-java#8460) - [@BaptisteFoy](https://github.com/BaptisteFoy)) - ✨ Probe trace endpoints with a valid payload of empty arrays ([#8414](DataDog/dd-trace-java#8414) - [@mcculls](https://github.com/mcculls)) - ✨ Add 1 minute fail-safe to JUL/JMX class-loading callback ([#8399](DataDog/dd-trace-java#8399) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate DSM injection calls to context-first APIs ([#8383](DataDog/dd-trace-java#8383) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Move continuation capture methods from scope to tracer ([#8371](DataDog/dd-trace-java#8371) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate context extraction calls to context-first APIs ([#8368](DataDog/dd-trace-java#8368) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Migrate context injection calls to context-first APIs ([#8358](DataDog/dd-trace-java#8358) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 💡 Support reading configurations from files ([#8338](DataDog/dd-trace-java#8338) - [@mtoffl01](https://github.com/mtoffl01)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - 🧹 Combine continuation implementations into one which supports multiple activations ([#8324](DataDog/dd-trace-java#8324) - [@mcculls](https://github.com/mcculls)) - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Remove old context propagation API ([#8271](DataDog/dd-trace-java#8271) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Instrumentations ##### AWS Lambda instrumentation - 🐛 Send error message and stack to Lambda extension ([#8417](DataDog/dd-trace-java#8417) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 💡 Inject trace context into AWS Step Functions input ([#7585](DataDog/dd-trace-java#7585) - [@DylanLovesCoffee](https://github.com/DylanLovesCoffee)) ##### Core Java language instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### gRPC instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) ##### Kafka instrumentation - ✨ Add messaging.destination.name tag to kafka integrations ([#8366](DataDog/dd-trace-java#8366) - [@rarguelloF](https://github.com/rarguelloF)) ##### Protocol Buffer instrumentation - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 108a0f86aa59ab4c938cbac0688dd4c19cb301fa

daniel-mohedano added 8 commits February 12, 2025 09:41

update is quarantined tag according to RFC
e3b7ad9

disabled tests
a5a065c

update TIA skip reason
58ec02a

add disabled smoke test
52acaed

update TIA skip reason in smoke tests
aac26fa

add test management check to skip instrumentations
8eb4cc1

include test management tests reset in smoke tests
8d340fb

join quarantine and disable maven smoke tests to test management
80a7054

daniel-mohedano added type: feature Enhancements and improvements comp: ci visibility Continuous Integration Visibility labels Feb 12, 2025

daniel-mohedano requested a review from nikita-tkachenko-datadog February 12, 2025 15:57

nikita-tkachenko-datadog approved these changes Feb 12, 2025
View reviewed changes

daniel-mohedano added 3 commits February 13, 2025 10:13

pr fixes
8c8d269

Merge branch 'master' into daniel.mohedano/disabled-tests
e75a57e

Merge branch 'master' into daniel.mohedano/disabled-tests
b98721e

daniel-mohedano marked this pull request as ready for review February 13, 2025 13:28

daniel-mohedano requested review from a team as code owners February 13, 2025 13:28

daniel-mohedano requested review from mcculls and tonyredondo February 13, 2025 13:28

daniel-mohedano merged commit dcf73eb into masterFeb 13, 2025

daniel-mohedano deleted the daniel.mohedano/disabled-tests branch February 13, 2025 13:29

github-actionsBot added this to the 1.47.0 milestone Feb 13, 2025

Navigation Menu

Uh oh!

Implement test disabling - #8377

Implement test disabling#8377
daniel-mohedano merged 11 commits into
masterfrom
daniel.mohedano/disabled-tests

daniel-mohedano commented Feb 12, 2025•
edited by atlassian Bot
Loading

Uh oh!

pr-commenterBot commented Feb 12, 2025•
edited
Loading

Uh oh!

nikita-tkachenko-datadog left a comment

Uh oh!

nikita-tkachenko-datadogFeb 12, 2025

Uh oh!

nikita-tkachenko-datadogFeb 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

daniel-mohedano commented Feb 12, 2025• edited by atlassian BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Contributor Checklist

Uh oh!

pr-commenterBot commented Feb 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

nikita-tkachenko-datadog left a comment

Choose a reason for hiding this comment

Uh oh!

nikita-tkachenko-datadogFeb 12, 2025

Choose a reason for hiding this comment

Uh oh!

nikita-tkachenko-datadogFeb 12, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniel-mohedano commented Feb 12, 2025•
edited by atlassian Bot
Loading

pr-commenterBot commented Feb 12, 2025•
edited
Loading