Implement quarantined tests tagging by nikita-tkachenko-datadog · Pull Request #8326 · DataDog/dd-trace-java

nikita-tkachenko-datadog · 2025-02-03T14:31:20Z

What Does This Do

Implements additional tags related to tests quarantine feature:

test.is_quarantined for tests cases that are quarantined
test.has_failed_all_retries for test cases that were retried multiple times and failed all of the executions

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1483

pr-commenter · 2025-02-03T15:12:37Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/quarantined-tests-tagging
git_commit_date	1738578316	1738592916
git_commit_sha	`9af5347`	`c80ed84`
release_version	1.47.0-SNAPSHOT~9af5347cda	1.47.0-SNAPSHOT~c80ed84ecc

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738595502	1738595502
ci_job_id	791309694	791309694
ci_pipeline_id	54719662	54719662
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-yhuzh6uy-project-304-concurrent-0-cl4wqp5e 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-yhuzh6uy-project-304-concurrent-0-cl4wqp5e 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.036 s) : 0, 1035731
Total [baseline] (10.496 s) : 0, 10495525
Agent [candidate] (1.041 s) : 0, 1040529
Total [candidate] (10.508 s) : 0, 10508084
section appsec
Agent [baseline] (1.182 s) : 0, 1182005
Total [baseline] (10.773 s) : 0, 10773059
Agent [candidate] (1.181 s) : 0, 1180887
Total [candidate] (10.749 s) : 0, 10749435
section iast
Agent [baseline] (1.168 s) : 0, 1168048
Total [baseline] (10.912 s) : 0, 10912426
Agent [candidate] (1.17 s) : 0, 1170445
Total [candidate] (10.961 s) : 0, 10961074
section profiling
Agent [baseline] (1.26 s) : 0, 1259908
Total [baseline] (10.897 s) : 0, 10896989
Agent [candidate] (1.264 s) : 0, 1263998
Total [candidate] (10.868 s) : 0, 10868094

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.036 s	-
Agent	appsec	1.182 s	146.274 ms (14.1%)
Agent	iast	1.168 s	132.317 ms (12.8%)
Agent	profiling	1.26 s	224.177 ms (21.6%)
Total	tracing	10.496 s	-
Total	appsec	10.773 s	277.534 ms (2.6%)
Total	iast	10.912 s	416.901 ms (4.0%)
Total	profiling	10.897 s	401.463 ms (3.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	appsec	1.181 s	140.358 ms (13.5%)
Agent	iast	1.17 s	129.916 ms (12.5%)
Agent	profiling	1.264 s	223.468 ms (21.5%)
Total	tracing	10.508 s	-
Total	appsec	10.749 s	241.351 ms (2.3%)
Total	iast	10.961 s	452.99 ms (4.3%)
Total	profiling	10.868 s	360.01 ms (3.4%)

gantt
title petclinic - break down per module: candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (712.644 ms) : 0, 712644
BytebuddyAgent [candidate] (716.057 ms) : 0, 716057
GlobalTracer [baseline] (241.844 ms) : 0, 241844
GlobalTracer [candidate] (240.871 ms) : 0, 240871
AppSec [baseline] (55.29 ms) : 0, 55290
AppSec [candidate] (55.454 ms) : 0, 55454
Remote Config [baseline] (714.278 µs) : 0, 714
Remote Config [candidate] (719.344 µs) : 0, 719
Telemetry [baseline] (10.082 ms) : 0, 10082
Telemetry [candidate] (12.237 ms) : 0, 12237
section appsec
BytebuddyAgent [baseline] (730.411 ms) : 0, 730411
BytebuddyAgent [candidate] (731.763 ms) : 0, 731763
GlobalTracer [baseline] (240.466 ms) : 0, 240466
GlobalTracer [candidate] (237.537 ms) : 0, 237537
AppSec [baseline] (176.253 ms) : 0, 176253
AppSec [candidate] (175.963 ms) : 0, 175963
Remote Config [baseline] (665.808 µs) : 0, 666
Remote Config [candidate] (661.538 µs) : 0, 662
Telemetry [baseline] (8.287 ms) : 0, 8287
Telemetry [candidate] (9.067 ms) : 0, 9067
IAST [baseline] (21.488 ms) : 0, 21488
IAST [candidate] (21.531 ms) : 0, 21531
section iast
BytebuddyAgent [baseline] (831.134 ms) : 0, 831134
BytebuddyAgent [candidate] (833.687 ms) : 0, 833687
GlobalTracer [baseline] (233.064 ms) : 0, 233064
GlobalTracer [candidate] (232.313 ms) : 0, 232313
AppSec [baseline] (54.553 ms) : 0, 54553
AppSec [candidate] (52.334 ms) : 0, 52334
Remote Config [baseline] (614.629 µs) : 0, 615
Remote Config [candidate] (621.459 µs) : 0, 621
Telemetry [baseline] (8.688 ms) : 0, 8688
Telemetry [candidate] (8.848 ms) : 0, 8848
IAST [baseline] (24.77 ms) : 0, 24770
IAST [candidate] (27.436 ms) : 0, 27436
section profiling
BytebuddyAgent [baseline] (704.753 ms) : 0, 704753
BytebuddyAgent [candidate] (709.882 ms) : 0, 709882
GlobalTracer [baseline] (352.42 ms) : 0, 352420
GlobalTracer [candidate] (352.362 ms) : 0, 352362
AppSec [baseline] (55.467 ms) : 0, 55467
AppSec [candidate] (54.375 ms) : 0, 54375
Remote Config [baseline] (701.715 µs) : 0, 702
Remote Config [candidate] (698.14 µs) : 0, 698
Telemetry [baseline] (8.93 ms) : 0, 8930
Telemetry [candidate] (8.953 ms) : 0, 8953
ProfilingAgent [baseline] (95.492 ms) : 0, 95492
ProfilingAgent [candidate] (95.306 ms) : 0, 95306
Profiling [baseline] (95.516 ms) : 0, 95516
Profiling [candidate] (95.33 ms) : 0, 95330

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.047 s) : 0, 1047222
Total [baseline] (8.668 s) : 0, 8667784
Agent [candidate] (1.037 s) : 0, 1036702
Total [candidate] (8.599 s) : 0, 8599286
section iast
Agent [baseline] (1.169 s) : 0, 1169078
Total [baseline] (9.25 s) : 0, 9250194
Agent [candidate] (1.175 s) : 0, 1175254
Total [candidate] (9.24 s) : 0, 9240317
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.172 s) : 0, 1171954
Total [baseline] (9.22 s) : 0, 9220375
Agent [candidate] (1.176 s) : 0, 1176241
Total [candidate] (9.182 s) : 0, 9182232
section iast_TELEMETRY_OFF
Agent [baseline] (1.166 s) : 0, 1165755
Total [baseline] (9.173 s) : 0, 9172934
Agent [candidate] (1.165 s) : 0, 1164923
Total [candidate] (9.232 s) : 0, 9231720

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	iast	1.169 s	121.857 ms (11.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.172 s	124.733 ms (11.9%)
Agent	iast_TELEMETRY_OFF	1.166 s	118.533 ms (11.3%)
Total	tracing	8.668 s	-
Total	iast	9.25 s	582.41 ms (6.7%)
Total	iast_HARDCODED_SECRET_DISABLED	9.22 s	552.592 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.173 s	505.15 ms (5.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	iast	1.175 s	138.552 ms (13.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.176 s	139.539 ms (13.5%)
Agent	iast_TELEMETRY_OFF	1.165 s	128.221 ms (12.4%)
Total	tracing	8.599 s	-
Total	iast	9.24 s	641.03 ms (7.5%)
Total	iast_HARDCODED_SECRET_DISABLED	9.182 s	582.945 ms (6.8%)
Total	iast_TELEMETRY_OFF	9.232 s	632.433 ms (7.4%)

gantt
title insecure-bank - break down per module: candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (720.886 ms) : 0, 720886
BytebuddyAgent [candidate] (713.042 ms) : 0, 713042
GlobalTracer [baseline] (242.708 ms) : 0, 242708
GlobalTracer [candidate] (239.833 ms) : 0, 239833
AppSec [baseline] (56.042 ms) : 0, 56042
AppSec [candidate] (55.122 ms) : 0, 55122
Remote Config [baseline] (710.415 µs) : 0, 710
Remote Config [candidate] (712.957 µs) : 0, 713
Telemetry [baseline] (11.448 ms) : 0, 11448
Telemetry [candidate] (12.794 ms) : 0, 12794
section iast
BytebuddyAgent [baseline] (832.171 ms) : 0, 832171
BytebuddyAgent [candidate] (838.644 ms) : 0, 838644
GlobalTracer [baseline] (233.454 ms) : 0, 233454
GlobalTracer [candidate] (232.218 ms) : 0, 232218
AppSec [baseline] (51.9 ms) : 0, 51900
AppSec [candidate] (54.083 ms) : 0, 54083
Remote Config [baseline] (601.772 µs) : 0, 602
Remote Config [candidate] (612.217 µs) : 0, 612
Telemetry [baseline] (8.615 ms) : 0, 8615
Telemetry [candidate] (8.734 ms) : 0, 8734
IAST [baseline] (27.067 ms) : 0, 27067
IAST [candidate] (25.685 ms) : 0, 25685
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (833.893 ms) : 0, 833893
BytebuddyAgent [candidate] (839.331 ms) : 0, 839331
GlobalTracer [baseline] (233.574 ms) : 0, 233574
GlobalTracer [candidate] (232.049 ms) : 0, 232049
AppSec [baseline] (53.985 ms) : 0, 53985
AppSec [candidate] (54.169 ms) : 0, 54169
Remote Config [baseline] (616.341 µs) : 0, 616
Remote Config [candidate] (621.214 µs) : 0, 621
Telemetry [baseline] (8.755 ms) : 0, 8755
Telemetry [candidate] (8.814 ms) : 0, 8814
IAST [baseline] (25.833 ms) : 0, 25833
IAST [candidate] (25.897 ms) : 0, 25897
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (829.568 ms) : 0, 829568
BytebuddyAgent [candidate] (830.541 ms) : 0, 830541
GlobalTracer [baseline] (233.244 ms) : 0, 233244
GlobalTracer [candidate] (230.781 ms) : 0, 230781
AppSec [baseline] (52.707 ms) : 0, 52707
AppSec [candidate] (53.1 ms) : 0, 53100
Remote Config [baseline] (612.999 µs) : 0, 613
Remote Config [candidate] (611.421 µs) : 0, 611
Telemetry [baseline] (8.548 ms) : 0, 8548
Telemetry [candidate] (8.627 ms) : 0, 8627
IAST [baseline] (25.849 ms) : 0, 25849
IAST [candidate] (26.065 ms) : 0, 26065

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-02-03T14:41:37	2025-02-03T14:48:41
git_branch	master	nikita-tkachenko/quarantined-tests-tagging
git_commit_date	1738578316	1738592916
git_commit_sha	`9af5347`	`c80ed84`
release_version	1.47.0-SNAPSHOT~9af5347cda	1.47.0-SNAPSHOT~c80ed84ecc
start_time	2025-02-03T14:41:23	2025-02-03T14:48:27

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738594478	1738594478
ci_job_id	791309695	791309695
ci_pipeline_id	54719662	54719662
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-kmuzcns6-project-304-concurrent-0-haicqptf 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-kmuzcns6-project-304-concurrent-0-haicqptf 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section baseline
no_agent (1.35 ms) : 1330, 1369
. : milestone, 1350,
appsec (1.773 ms) : 1750, 1796
. : milestone, 1773,
appsec_no_iast (1.74 ms) : 1715, 1765
. : milestone, 1740,
iast (1.516 ms) : 1492, 1540
. : milestone, 1516,
profiling (1.547 ms) : 1523, 1572
. : milestone, 1547,
tracing (1.483 ms) : 1458, 1508
. : milestone, 1483,
section candidate
no_agent (1.351 ms) : 1331, 1370
. : milestone, 1351,
appsec (1.742 ms) : 1718, 1765
. : milestone, 1742,
appsec_no_iast (1.732 ms) : 1707, 1757
. : milestone, 1732,
iast (1.508 ms) : 1484, 1531
. : milestone, 1508,
profiling (1.533 ms) : 1508, 1558
. : milestone, 1533,
tracing (1.49 ms) : 1466, 1514
. : milestone, 1490,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.35 ms [1.33 ms, 1.369 ms]	-
appsec	1.773 ms [1.75 ms, 1.796 ms]	423.13 µs (31.4%)
appsec_no_iast	1.74 ms [1.715 ms, 1.765 ms]	390.771 µs (29.0%)
iast	1.516 ms [1.492 ms, 1.54 ms]	166.728 µs (12.4%)
profiling	1.547 ms [1.523 ms, 1.572 ms]	197.896 µs (14.7%)
tracing	1.483 ms [1.458 ms, 1.508 ms]	133.21 µs (9.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.351 ms [1.331 ms, 1.37 ms]	-
appsec	1.742 ms [1.718 ms, 1.765 ms]	391.254 µs (29.0%)
appsec_no_iast	1.732 ms [1.707 ms, 1.757 ms]	381.445 µs (28.2%)
iast	1.508 ms [1.484 ms, 1.531 ms]	157.002 µs (11.6%)
profiling	1.533 ms [1.508 ms, 1.558 ms]	182.727 µs (13.5%)
tracing	1.49 ms [1.466 ms, 1.514 ms]	139.45 µs (10.3%)

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section baseline
no_agent (377.07 µs) : 358, 397
. : milestone, 377,
iast (508.068 µs) : 487, 530
. : milestone, 508,
iast_FULL (737.896 µs) : 716, 760
. : milestone, 738,
iast_GLOBAL (556.884 µs) : 534, 579
. : milestone, 557,
iast_HARDCODED_SECRET_DISABLED (506.351 µs) : 484, 529
. : milestone, 506,
iast_INACTIVE (457.25 µs) : 435, 479
. : milestone, 457,
iast_TELEMETRY_OFF (493.415 µs) : 471, 516
. : milestone, 493,
tracing (455.249 µs) : 434, 476
. : milestone, 455,
section candidate
no_agent (376.016 µs) : 356, 396
. : milestone, 376,
iast (499.651 µs) : 478, 521
. : milestone, 500,
iast_FULL (738.0 µs) : 716, 760
. : milestone, 738,
iast_GLOBAL (549.246 µs) : 528, 571
. : milestone, 549,
iast_HARDCODED_SECRET_DISABLED (509.28 µs) : 487, 532
. : milestone, 509,
iast_INACTIVE (458.829 µs) : 437, 480
. : milestone, 459,
iast_TELEMETRY_OFF (496.154 µs) : 473, 520
. : milestone, 496,
tracing (452.119 µs) : 431, 473
. : milestone, 452,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	377.07 µs [357.527 µs, 396.612 µs]	-
iast	508.068 µs [486.505 µs, 529.631 µs]	130.998 µs (34.7%)
iast_FULL	737.896 µs [715.867 µs, 759.924 µs]	360.826 µs (95.7%)
iast_GLOBAL	556.884 µs [534.46 µs, 579.308 µs]	179.815 µs (47.7%)
iast_HARDCODED_SECRET_DISABLED	506.351 µs [483.929 µs, 528.772 µs]	129.281 µs (34.3%)
iast_INACTIVE	457.25 µs [435.339 µs, 479.162 µs]	80.18 µs (21.3%)
iast_TELEMETRY_OFF	493.415 µs [470.921 µs, 515.908 µs]	116.345 µs (30.9%)
tracing	455.249 µs [434.136 µs, 476.363 µs]	78.179 µs (20.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	376.016 µs [356.266 µs, 395.766 µs]	-
iast	499.651 µs [478.246 µs, 521.057 µs]	123.635 µs (32.9%)
iast_FULL	738.0 µs [716.159 µs, 759.841 µs]	361.983 µs (96.3%)
iast_GLOBAL	549.246 µs [527.525 µs, 570.967 µs]	173.23 µs (46.1%)
iast_HARDCODED_SECRET_DISABLED	509.28 µs [486.957 µs, 531.602 µs]	133.263 µs (35.4%)
iast_INACTIVE	458.829 µs [437.175 µs, 480.483 µs]	82.813 µs (22.0%)
iast_TELEMETRY_OFF	496.154 µs [472.579 µs, 519.729 µs]	120.138 µs (32.0%)
tracing	452.119 µs [431.096 µs, 473.141 µs]	76.102 µs (20.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/quarantined-tests-tagging
git_commit_date	1738578316	1738592916
git_commit_sha	`9af5347`	`c80ed84`
release_version	1.47.0-SNAPSHOT~9af5347cda	1.47.0-SNAPSHOT~c80ed84ecc

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1738595042	1738595042
ci_job_id	791309696	791309696
ci_pipeline_id	54719662	54719662
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-rzaamrpw-project-304-concurrent-0-iv5264yf 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-rzaamrpw-project-304-concurrent-0-iv5264yf 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section baseline
no_agent (14.94 s) : 14940000, 14940000
. : milestone, 14940000,
appsec (15.321 s) : 15321000, 15321000
. : milestone, 15321000,
iast (19.101 s) : 19101000, 19101000
. : milestone, 19101000,
iast_GLOBAL (18.319 s) : 18319000, 18319000
. : milestone, 18319000,
profiling (15.089 s) : 15089000, 15089000
. : milestone, 15089000,
tracing (14.819 s) : 14819000, 14819000
. : milestone, 14819000,
section candidate
no_agent (15.103 s) : 15103000, 15103000
. : milestone, 15103000,
appsec (15.098 s) : 15098000, 15098000
. : milestone, 15098000,
iast (18.511 s) : 18511000, 18511000
. : milestone, 18511000,
iast_GLOBAL (18.185 s) : 18185000, 18185000
. : milestone, 18185000,
profiling (15.169 s) : 15169000, 15169000
. : milestone, 15169000,
tracing (14.886 s) : 14886000, 14886000
. : milestone, 14886000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.94 s [14.94 s, 14.94 s]	-
appsec	15.321 s [15.321 s, 15.321 s]	381.0 ms (2.6%)
iast	19.101 s [19.101 s, 19.101 s]	4.161 s (27.9%)
iast_GLOBAL	18.319 s [18.319 s, 18.319 s]	3.379 s (22.6%)
profiling	15.089 s [15.089 s, 15.089 s]	149.0 ms (1.0%)
tracing	14.819 s [14.819 s, 14.819 s]	-121.0 ms (-0.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.103 s [15.103 s, 15.103 s]	-
appsec	15.098 s [15.098 s, 15.098 s]	-5.0 ms (-0.0%)
iast	18.511 s [18.511 s, 18.511 s]	3.408 s (22.6%)
iast_GLOBAL	18.185 s [18.185 s, 18.185 s]	3.082 s (20.4%)
profiling	15.169 s [15.169 s, 15.169 s]	66.0 ms (0.4%)
tracing	14.886 s [14.886 s, 14.886 s]	-217.0 ms (-1.4%)

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~c80ed84ecc, baseline=1.47.0-SNAPSHOT~9af5347cda
dateFormat X
axisFormat %s
section baseline
no_agent (1.477 ms) : 1466, 1489
. : milestone, 1477,
appsec (2.365 ms) : 2322, 2408
. : milestone, 2365,
iast (2.113 ms) : 2058, 2167
. : milestone, 2113,
iast_GLOBAL (2.17 ms) : 2115, 2225
. : milestone, 2170,
profiling (2.003 ms) : 1958, 2048
. : milestone, 2003,
tracing (1.957 ms) : 1915, 1999
. : milestone, 1957,
section candidate
no_agent (1.481 ms) : 1469, 1492
. : milestone, 1481,
appsec (2.366 ms) : 2323, 2409
. : milestone, 2366,
iast (2.129 ms) : 2074, 2183
. : milestone, 2129,
iast_GLOBAL (2.161 ms) : 2106, 2216
. : milestone, 2161,
profiling (1.971 ms) : 1927, 2014
. : milestone, 1971,
tracing (1.953 ms) : 1911, 1994
. : milestone, 1953,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.489 ms]	-
appsec	2.365 ms [2.322 ms, 2.408 ms]	888.009 µs (60.1%)
iast	2.113 ms [2.058 ms, 2.167 ms]	635.49 µs (43.0%)
iast_GLOBAL	2.17 ms [2.115 ms, 2.225 ms]	693.042 µs (46.9%)
profiling	2.003 ms [1.958 ms, 2.048 ms]	525.914 µs (35.6%)
tracing	1.957 ms [1.915 ms, 1.999 ms]	480.14 µs (32.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.481 ms [1.469 ms, 1.492 ms]	-
appsec	2.366 ms [2.323 ms, 2.409 ms]	885.418 µs (59.8%)
iast	2.129 ms [2.074 ms, 2.183 ms]	648.241 µs (43.8%)
iast_GLOBAL	2.161 ms [2.106 ms, 2.216 ms]	680.369 µs (46.0%)
profiling	1.971 ms [1.927 ms, 2.014 ms]	490.126 µs (33.1%)
tracing	1.953 ms [1.911 ms, 1.994 ms]	471.937 µs (31.9%)

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | --- ### Release Notes <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.47.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.47.0): 1.47.0 ##### Components ##### Application Security Management (IAST) - 🐛 Exclude com.stripe.net.HttpURLConnectionClient to solve IAST SSRF vulnerability false positives ([#8483](DataDog/dd-trace-java#8483) - [@jandro996](https://github.com/jandro996)) - 🐛 Add exclusion to solve IAST weak randomness vulnerability false positives ([#8462](DataDog/dd-trace-java#8462) - [@jandro996](https://github.com/jandro996)) - ✨ Fix weak randomness false positive in Kafka client ([#8408](DataDog/dd-trace-java#8408) - [@smola](https://github.com/smola)) - ✨ Fix location for SSRF with Kong Unirest ([#8407](DataDog/dd-trace-java#8407) - [@smola](https://github.com/smola)) - ✨ Exclude IBM Instana from IAST ([#8406](DataDog/dd-trace-java#8406) - [@smola](https://github.com/smola)) - 🐛 Fix org.json iast instrumentation test for latest dependency ([#8347](DataDog/dd-trace-java#8347) - [@jandro996](https://github.com/jandro996)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) - ✨ Address cookie vulnerability cardinality issues ([#8210](DataDog/dd-trace-java#8210) - [@jandro996](https://github.com/jandro996)) - ✨ Email HTML Injection detection in IAST ([#8205](DataDog/dd-trace-java#8205) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Application Security Management (WAF) - 🐛✨ Ensure usr.exists tag is not overridden when UsernameNotFoundException is thrown ([#8376](DataDog/dd-trace-java#8376) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛✨ Ensure usr.exists tag is not overridden by auto instrumentation ([#8374](DataDog/dd-trace-java#8374) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update appsec metrics with event_rules_version tag ([#8354](DataDog/dd-trace-java#8354) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) - ✨ Improve ASM support in vert.x 5.0 ([#8285](DataDog/dd-trace-java#8285) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update metrics: appsec.waf.updates and appsec.waf.init ([#8280](DataDog/dd-trace-java#8280) - [@Mariovido](https://github.com/Mariovido)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) ##### Build & Tooling - 🐛 Do not generate Muzzle references for primitive arrays in method body ([#8361](DataDog/dd-trace-java#8361) - [@amarziali](https://github.com/amarziali)) - 📖 Improve dev env setup documentation for Windows ([#8180](DataDog/dd-trace-java#8180) - [@lucaspimentel](https://github.com/lucaspimentel)) ##### Continuous Integration Visibility - ✨ Add support for skip-EFD tagging ([#8487](DataDog/dd-trace-java#8487) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix an NPE in Gradle Android instrumentation ([#8484](DataDog/dd-trace-java#8484) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Consider modified tests when applying fail-fast tests ordering ([#8474](DataDog/dd-trace-java#8474) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests reordering for TestNG ([#8467](DataDog/dd-trace-java#8467) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Gradle Launcher instrumentation to not interfere with Gradle Test Kit ([#8465](DataDog/dd-trace-java#8465) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Use separate TestEventHandlers per framework in CI Vis instrumentations ([#8451](DataDog/dd-trace-java#8451) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Remove warning log when JUnit 4 test method cannot be retrieved ([#8445](DataDog/dd-trace-java#8445) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Scalatest tracing for tests that are reported asynchronously ([#8444](DataDog/dd-trace-java#8444) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement attempt to fix tests ([#8393](DataDog/dd-trace-java#8393) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement test disabling ([#8377](DataDog/dd-trace-java#8377) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Update CODEOWNERS parser to not log errors on comments with leading whitespace ([#8349](DataDog/dd-trace-java#8349) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Request Test Management tests list ([#8345](DataDog/dd-trace-java#8345) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Receive test management settings from CIVis settings request ([#8331](DataDog/dd-trace-java#8331) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement quarantined tests tagging ([#8326](DataDog/dd-trace-java#8326) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests quarantining ([#8320](DataDog/dd-trace-java#8320) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add tag to specify if the user is setting DD_SERVICE ([#8318](DataDog/dd-trace-java#8318) - [@daniel-mohedano](https://github.com/daniel-mohedano)) ##### Crash tracking - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Use Java home of the crashed process to launch crash uploader ([#8348](DataDog/dd-trace-java#8348) - [@jbachorik](https://github.com/jbachorik)) ##### Data Streams Monitoring - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix service name overrides in consumers ([#8387](DataDog/dd-trace-java#8387) - [@piochelepiotr](https://github.com/piochelepiotr)) ##### Database Monitoring - ✨ Add DBMTracePreparedStatements to tracer configuration log ([#8508](DataDog/dd-trace-java#8508) - [@cecile75](https://github.com/cecile75)) ##### Dynamic Instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Fix Exception Replay with Lambda proxy classes ([#8452](DataDog/dd-trace-java#8452) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add support for scanning jar from loaded class ([#8370](DataDog/dd-trace-java#8370) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Disable capture of entry values ([#8369](DataDog/dd-trace-java#8369) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix CodeOrigin for `@Trace` annotation ([#8344](DataDog/dd-trace-java#8344) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix equals/hashCode for CodeOrigin probe ([#8319](DataDog/dd-trace-java#8319) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### Metrics - ✨ Create metric: appsec.waf.error ([#8381](DataDog/dd-trace-java#8381) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Create metric: appsec.rasp.error ([#8364](DataDog/dd-trace-java#8364) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Profiling - ✨ Bump ddprof library to 1.22.0 ([#8463](DataDog/dd-trace-java#8463) - [@jbachorik](https://github.com/jbachorik)) - IBM J9 8u361 corresponds to OpenJDK 8u362 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#187 - Fix compatibility with musl libc 1.2.4 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#189 - Modify version extraction by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#179 - Do not write null values to jvminfo event by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#184 - Productize VMStructs-based stack walker by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#177 - A few minor downport issues by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#180 - Enable ASGCT by default on fairly safe J9 JDK versions by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#181 - 🐛 Exclude OrderedThreadPoolExecutor from queue-time measurements ([#8456](DataDog/dd-trace-java#8456) - [@jbachorik](https://github.com/jbachorik)) - ✨ Record JVM info on JVMs without JFR ([#8431](DataDog/dd-trace-java#8431) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Actually use CleanupTask in TempLocationManager ([#8420](DataDog/dd-trace-java#8420) - [@mcculls](https://github.com/mcculls)) - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Adjust JFR checks for J9 ([#8405](DataDog/dd-trace-java#8405) - [@jbachorik](https://github.com/jbachorik)) - 🧹 Disable smap RSS parsing by default ([#8342](DataDog/dd-trace-java#8342) - [@MattAlp](https://github.com/MattAlp)) ##### Telemetry - 🐛 Add support for JBoss jar:file format to DependencyResolver ([#8428](DataDog/dd-trace-java#8428) - [@jandro996](https://github.com/jandro996)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) ##### Trace context propagation - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - 🐛 Fix Stable Config telemetry source names ([#8460](DataDog/dd-trace-java#8460) - [@BaptisteFoy](https://github.com/BaptisteFoy)) - ✨ Probe trace endpoints with a valid payload of empty arrays ([#8414](DataDog/dd-trace-java#8414) - [@mcculls](https://github.com/mcculls)) - ✨ Add 1 minute fail-safe to JUL/JMX class-loading callback ([#8399](DataDog/dd-trace-java#8399) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate DSM injection calls to context-first APIs ([#8383](DataDog/dd-trace-java#8383) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Move continuation capture methods from scope to tracer ([#8371](DataDog/dd-trace-java#8371) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate context extraction calls to context-first APIs ([#8368](DataDog/dd-trace-java#8368) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Migrate context injection calls to context-first APIs ([#8358](DataDog/dd-trace-java#8358) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 💡 Support reading configurations from files ([#8338](DataDog/dd-trace-java#8338) - [@mtoffl01](https://github.com/mtoffl01)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - 🧹 Combine continuation implementations into one which supports multiple activations ([#8324](DataDog/dd-trace-java#8324) - [@mcculls](https://github.com/mcculls)) - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Remove old context propagation API ([#8271](DataDog/dd-trace-java#8271) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Instrumentations ##### AWS Lambda instrumentation - 🐛 Send error message and stack to Lambda extension ([#8417](DataDog/dd-trace-java#8417) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 💡 Inject trace context into AWS Step Functions input ([#7585](DataDog/dd-trace-java#7585) - [@DylanLovesCoffee](https://github.com/DylanLovesCoffee)) ##### Core Java language instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### gRPC instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) ##### Kafka instrumentation - ✨ Add messaging.destination.name tag to kafka integrations ([#8366](DataDog/dd-trace-java#8366) - [@rarguelloF](https://github.com/rarguelloF)) ##### Protocol Buffer instrumentation - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 108a0f86aa59ab4c938cbac0688dd4c19cb301fa

nikita-tkachenko-datadog added 5 commits January 31, 2025 16:49

Implement quarantined tests tagging
7dfe567

Add is_quarantined tag to test fixtures
0c58002

Has failed all retries tagging
9ec150b

Fix TestNG retry listener ordering problem
f5c4b8f

Update test fixtures
c80ed84

nikita-tkachenko-datadog added type: feature Enhancements and improvements comp: ci visibility Continuous Integration Visibility labels Feb 3, 2025

nikita-tkachenko-datadog changed the title ~~Nikita tkachenko/quarantined tests tagging~~Implement quarantined tests taggingFeb 3, 2025

daniel-mohedano approved these changes Feb 3, 2025
View reviewed changes

nikita-tkachenko-datadog marked this pull request as ready for review February 3, 2025 15:30

nikita-tkachenko-datadog requested review from a team as code owners February 3, 2025 15:30

nikita-tkachenko-datadog requested review from nayeem-kamal and tonyredondo February 3, 2025 15:30

nikita-tkachenko-datadog merged commit 38b8211 into masterFeb 3, 2025

nikita-tkachenko-datadog deleted the nikita-tkachenko/quarantined-tests-tagging branch February 3, 2025 15:37

github-actionsBot added this to the 1.47.0 milestone Feb 3, 2025

Navigation Menu

Uh oh!

Implement quarantined tests tagging - #8326

Implement quarantined tests tagging#8326
nikita-tkachenko-datadog merged 5 commits into
masterfrom
nikita-tkachenko/quarantined-tests-tagging

nikita-tkachenko-datadog commented Feb 3, 2025•
edited
Loading

Uh oh!

pr-commenterBot commented Feb 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nikita-tkachenko-datadog commented Feb 3, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Contributor Checklist

Uh oh!

pr-commenterBot commented Feb 3, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nikita-tkachenko-datadog commented Feb 3, 2025•
edited
Loading