Fix test selection logic with coverage by smola · Pull Request #7467 · DataDog/dd-trace-java

smola · 2024-08-19T06:53:03Z

What Does This Do

The way we called jacoco tasks for gradle will trigger tests outside those selected by :baseTests at al. Remove explicit calls to these tasks from CI, and use a -PcheckCoverage property instead. When combined with :baseTests, it will run jacoco tasks only for the selected tests.

Remove legacy properties -PskipInstTests, -PskipSmokeTests, etc. These are now obsolete, and together with the above fix, they do not serve any purpose in CI anymore.

:debuggerTest have test coverage always enabled. This behavior is preserved.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Squash your commits prior merging or merge using GitHub's Squash and merge
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
~~[ ] Update the public documentation in case of new configuration flag or behavior~~

pr-commenter · 2024-08-19T07:12:45Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1724065531	1724065910
end_time	2024-08-19T11:06:46	2024-08-19T11:13:03
git_branch	master	smola/fix-ci-test-selection
git_commit_sha	`5ae921b`	`e2947e2`
start_time	2024-08-19T11:05:32	2024-08-19T11:11:51

See matching parameters

	Baseline	Candidate
ci_job_id	608592932	608592932
ci_pipeline_id	42101233	42101233
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1724064974	1724064974

Summary

Found 0 performance improvements and 5 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:loop	worse [+705.320µs; +736.787µs] or [+6.844%; +7.149%]	worse [+700.083µs; +754.943µs] or [+6.685%; +7.209%]	worse [+679.277µs; +784.170µs] or [+6.432%; +7.426%]	worse [+612.212µs; +1016.700µs] or [+5.598%; +9.296%]	worse [-6.531op/s; -5.157op/s] or [-6.923%; -5.466%]

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-23.625µs; +11.839µs] or [-9.120%; +4.570%]	unstable [-38.597µs; +18.283µs] or [-13.088%; +6.200%]	unstable [-48.948µs; +29.822µs] or [-15.942%; +9.713%]	unstable [-94.509µs; +83.543µs] or [-14.673%; +12.970%]	same
scenario:basic	unsure [-9.597µs; -0.429µs] or [-3.515%; -0.157%]	same	same	unstable [-15.873µs; +52.834µs] or [-2.799%; +9.316%]	unstable [-84.689op/s; +219.641op/s] or [-3.303%; +8.566%]

Request duration reports for reports

gantt
title reports - request duration [CI 0.99] : candidate=None, baseline=None
dateFormat X
axisFormat %s
section baseline
noprobe (294.905 µs) : 265, 325
. : milestone, 295,
basic (299.575 µs) : 293, 306
. : milestone, 300,
loop (10.473 ms) : 10446, 10500
. : milestone, 10473,
section candidate
noprobe (284.748 µs) : 261, 309
. : milestone, 285,
basic (292.64 µs) : 285, 300
. : milestone, 293,
loop (11.2 ms) : 11177, 11224
. : milestone, 11200,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	294.905 µs [264.784 µs, 325.025 µs]
basic	299.575 µs [292.866 µs, 306.285 µs]
loop	10.473 ms [10.446 ms, 10.5 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	284.748 µs [260.97 µs, 308.526 µs]
basic	292.64 µs [284.936 µs, 300.345 µs]
loop	11.2 ms [11.177 ms, 11.224 ms]

pr-commenter · 2024-08-19T07:30:52Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/fix-ci-test-selection
git_commit_date	1724059733	1724064974
git_commit_sha	`5ae921b`	`e2947e2`
release_version	1.39.0-SNAPSHOT~5ae921b9c4	1.39.0-SNAPSHOT~e2947e25fc

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1724067239	1724067239
ci_job_id	608592922	608592922
ci_pipeline_id	42101233	42101233
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 15 unstable metrics.

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049442
Total [baseline] (8.509 s) : 0, 8509273
Agent [candidate] (1.046 s) : 0, 1046331
Total [candidate] (8.49 s) : 0, 8489668
section iast
Agent [baseline] (1.204 s) : 0, 1203583
Total [baseline] (9.087 s) : 0, 9087453
Agent [candidate] (1.178 s) : 0, 1177975
Total [candidate] (8.998 s) : 0, 8997864
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.18 s) : 0, 1179900
Total [baseline] (8.979 s) : 0, 8978967
Agent [candidate] (1.178 s) : 0, 1178266
Total [candidate] (8.97 s) : 0, 8970071
section iast_TELEMETRY_OFF
Agent [baseline] (1.172 s) : 0, 1172376
Total [baseline] (8.986 s) : 0, 8986349
Agent [candidate] (1.172 s) : 0, 1172237
Total [candidate] (9.03 s) : 0, 9029906

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	iast	1.204 s	154.141 ms (14.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.18 s	130.457 ms (12.4%)
Agent	iast_TELEMETRY_OFF	1.172 s	122.934 ms (11.7%)
Total	tracing	8.509 s	-
Total	iast	9.087 s	578.181 ms (6.8%)
Total	iast_HARDCODED_SECRET_DISABLED	8.979 s	469.695 ms (5.5%)
Total	iast_TELEMETRY_OFF	8.986 s	477.077 ms (5.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.046 s	-
Agent	iast	1.178 s	131.644 ms (12.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.178 s	131.935 ms (12.6%)
Agent	iast_TELEMETRY_OFF	1.172 s	125.906 ms (12.0%)
Total	tracing	8.49 s	-
Total	iast	8.998 s	508.195 ms (6.0%)
Total	iast_HARDCODED_SECRET_DISABLED	8.97 s	480.403 ms (5.7%)
Total	iast_TELEMETRY_OFF	9.03 s	540.237 ms (6.4%)

gantt
title insecure-bank - break down per module: candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (668.919 ms) : 0, 668919
BytebuddyAgent [candidate] (667.099 ms) : 0, 667099
GlobalTracer [baseline] (307.403 ms) : 0, 307403
GlobalTracer [candidate] (306.393 ms) : 0, 306393
AppSec [baseline] (51.416 ms) : 0, 51416
AppSec [candidate] (51.3 ms) : 0, 51300
Remote Config [baseline] (686.935 µs) : 0, 687
Remote Config [candidate] (676.789 µs) : 0, 677
Telemetry [baseline] (7.58 ms) : 0, 7580
Telemetry [candidate] (7.467 ms) : 0, 7467
section iast
BytebuddyAgent [baseline] (801.689 ms) : 0, 801689
BytebuddyAgent [candidate] (783.506 ms) : 0, 783506
GlobalTracer [baseline] (302.762 ms) : 0, 302762
GlobalTracer [candidate] (297.099 ms) : 0, 297099
AppSec [baseline] (54.596 ms) : 0, 54596
AppSec [candidate] (52.113 ms) : 0, 52113
Remote Config [baseline] (595.986 µs) : 0, 596
Remote Config [candidate] (577.055 µs) : 0, 577
Telemetry [baseline] (8.155 ms) : 0, 8155
Telemetry [candidate] (9.575 ms) : 0, 9575
IAST [baseline] (22.047 ms) : 0, 22047
IAST [candidate] (21.589 ms) : 0, 21589
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (783.937 ms) : 0, 783937
BytebuddyAgent [candidate] (782.495 ms) : 0, 782495
GlobalTracer [baseline] (298.184 ms) : 0, 298184
GlobalTracer [candidate] (297.435 ms) : 0, 297435
AppSec [baseline] (50.446 ms) : 0, 50446
AppSec [candidate] (48.012 ms) : 0, 48012
Remote Config [baseline] (583.467 µs) : 0, 583
Remote Config [candidate] (587.798 µs) : 0, 588
Telemetry [baseline] (8.808 ms) : 0, 8808
Telemetry [candidate] (11.867 ms) : 0, 11867
IAST [baseline] (24.387 ms) : 0, 24387
IAST [candidate] (24.333 ms) : 0, 24333
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (779.715 ms) : 0, 779715
BytebuddyAgent [candidate] (779.094 ms) : 0, 779094
GlobalTracer [baseline] (296.948 ms) : 0, 296948
GlobalTracer [candidate] (297.211 ms) : 0, 297211
AppSec [baseline] (52.499 ms) : 0, 52499
AppSec [candidate] (53.202 ms) : 0, 53202
Remote Config [baseline] (619.045 µs) : 0, 619
Remote Config [candidate] (613.659 µs) : 0, 614
Telemetry [baseline] (7.003 ms) : 0, 7003
Telemetry [candidate] (8.029 ms) : 0, 8029
IAST [baseline] (22.077 ms) : 0, 22077
IAST [candidate] (20.602 ms) : 0, 20602

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.047 s) : 0, 1046618
Total [baseline] (10.364 s) : 0, 10363731
Agent [candidate] (1.049 s) : 0, 1048749
Total [candidate] (10.416 s) : 0, 10416039
section appsec
Agent [baseline] (1.17 s) : 0, 1169884
Total [baseline] (10.452 s) : 0, 10451601
Agent [candidate] (1.181 s) : 0, 1181121
Total [candidate] (10.469 s) : 0, 10469254
section iast
Agent [baseline] (1.178 s) : 0, 1177822
Total [baseline] (10.791 s) : 0, 10790991
Agent [candidate] (1.18 s) : 0, 1180345
Total [candidate] (10.822 s) : 0, 10821765
section profiling
Agent [baseline] (1.245 s) : 0, 1244828
Total [baseline] (10.578 s) : 0, 10578334
Agent [candidate] (1.249 s) : 0, 1249215
Total [candidate] (10.578 s) : 0, 10578281

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	appsec	1.17 s	123.267 ms (11.8%)
Agent	iast	1.178 s	131.204 ms (12.5%)
Agent	profiling	1.245 s	198.211 ms (18.9%)
Total	tracing	10.364 s	-
Total	appsec	10.452 s	87.87 ms (0.8%)
Total	iast	10.791 s	427.26 ms (4.1%)
Total	profiling	10.578 s	214.603 ms (2.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	appsec	1.181 s	132.372 ms (12.6%)
Agent	iast	1.18 s	131.596 ms (12.5%)
Agent	profiling	1.249 s	200.467 ms (19.1%)
Total	tracing	10.416 s	-
Total	appsec	10.469 s	53.215 ms (0.5%)
Total	iast	10.822 s	405.726 ms (3.9%)
Total	profiling	10.578 s	162.242 ms (1.6%)

gantt
title petclinic - break down per module: candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (667.687 ms) : 0, 667687
BytebuddyAgent [candidate] (668.939 ms) : 0, 668939
GlobalTracer [baseline] (306.364 ms) : 0, 306364
GlobalTracer [candidate] (307.219 ms) : 0, 307219
AppSec [baseline] (51.021 ms) : 0, 51021
AppSec [candidate] (50.992 ms) : 0, 50992
Remote Config [baseline] (679.975 µs) : 0, 680
Remote Config [candidate] (678.595 µs) : 0, 679
Telemetry [baseline] (7.459 ms) : 0, 7459
Telemetry [candidate] (7.48 ms) : 0, 7480
section appsec
BytebuddyAgent [baseline] (678.695 ms) : 0, 678695
BytebuddyAgent [candidate] (686.377 ms) : 0, 686377
GlobalTracer [baseline] (300.292 ms) : 0, 300292
GlobalTracer [candidate] (303.686 ms) : 0, 303686
AppSec [baseline] (157.663 ms) : 0, 157663
AppSec [candidate] (157.985 ms) : 0, 157985
IAST [baseline] (21.493 ms) : 0, 21493
IAST [candidate] (20.91 ms) : 0, 20910
Remote Config [baseline] (603.477 µs) : 0, 603
Remote Config [candidate] (601.327 µs) : 0, 601
Telemetry [baseline] (8.401 ms) : 0, 8401
Telemetry [candidate] (8.398 ms) : 0, 8398
section iast
BytebuddyAgent [baseline] (783.328 ms) : 0, 783328
BytebuddyAgent [candidate] (784.449 ms) : 0, 784449
GlobalTracer [baseline] (296.519 ms) : 0, 296519
GlobalTracer [candidate] (297.491 ms) : 0, 297491
AppSec [baseline] (53.345 ms) : 0, 53345
AppSec [candidate] (50.126 ms) : 0, 50126
IAST [baseline] (23.465 ms) : 0, 23465
IAST [candidate] (26.0 ms) : 0, 26000
Remote Config [baseline] (583.3 µs) : 0, 583
Remote Config [candidate] (623.704 µs) : 0, 624
Telemetry [baseline] (7.074 ms) : 0, 7074
Telemetry [candidate] (8.124 ms) : 0, 8124
section profiling
BytebuddyAgent [baseline] (662.711 ms) : 0, 662711
BytebuddyAgent [candidate] (664.831 ms) : 0, 664831
GlobalTracer [baseline] (390.468 ms) : 0, 390468
GlobalTracer [candidate] (391.784 ms) : 0, 391784
AppSec [baseline] (52.362 ms) : 0, 52362
AppSec [candidate] (52.18 ms) : 0, 52180
Remote Config [baseline] (687.983 µs) : 0, 688
Remote Config [candidate] (698.967 µs) : 0, 699
Telemetry [baseline] (7.429 ms) : 0, 7429
Telemetry [candidate] (7.441 ms) : 0, 7441
ProfilingAgent [baseline] (94.149 ms) : 0, 94149
ProfilingAgent [candidate] (95.015 ms) : 0, 95015
Profiling [baseline] (94.175 ms) : 0, 94175
Profiling [candidate] (95.039 ms) : 0, 95039

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-08-19T11:04:41	2024-08-19T11:11:29
git_branch	master	smola/fix-ci-test-selection
git_commit_date	1724059733	1724064974
git_commit_sha	`5ae921b`	`e2947e2`
release_version	1.39.0-SNAPSHOT~5ae921b9c4	1.39.0-SNAPSHOT~e2947e25fc
start_time	2024-08-19T11:04:28	2024-08-19T11:11:16

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1724066233	1724066233
ci_job_id	608592923	608592923
ci_pipeline_id	42101233	42101233
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section baseline
no_agent (370.381 µs) : 351, 390
. : milestone, 370,
iast (483.025 µs) : 461, 505
. : milestone, 483,
iast_FULL (547.155 µs) : 525, 570
. : milestone, 547,
iast_GLOBAL (507.452 µs) : 485, 530
. : milestone, 507,
iast_HARDCODED_SECRET_DISABLED (479.544 µs) : 459, 500
. : milestone, 480,
iast_INACTIVE (457.447 µs) : 435, 480
. : milestone, 457,
iast_TELEMETRY_OFF (471.85 µs) : 451, 493
. : milestone, 472,
tracing (442.305 µs) : 422, 463
. : milestone, 442,
section candidate
no_agent (364.334 µs) : 344, 384
. : milestone, 364,
iast (480.078 µs) : 459, 501
. : milestone, 480,
iast_FULL (548.575 µs) : 527, 571
. : milestone, 549,
iast_GLOBAL (505.706 µs) : 484, 528
. : milestone, 506,
iast_HARDCODED_SECRET_DISABLED (483.121 µs) : 461, 505
. : milestone, 483,
iast_INACTIVE (454.87 µs) : 433, 477
. : milestone, 455,
iast_TELEMETRY_OFF (466.739 µs) : 446, 488
. : milestone, 467,
tracing (444.594 µs) : 424, 466
. : milestone, 445,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	370.381 µs [351.239 µs, 389.522 µs]	-
iast	483.025 µs [461.253 µs, 504.796 µs]	112.644 µs (30.4%)
iast_FULL	547.155 µs [524.699 µs, 569.611 µs]	176.774 µs (47.7%)
iast_GLOBAL	507.452 µs [485.326 µs, 529.577 µs]	137.071 µs (37.0%)
iast_HARDCODED_SECRET_DISABLED	479.544 µs [458.633 µs, 500.455 µs]	109.164 µs (29.5%)
iast_INACTIVE	457.447 µs [434.937 µs, 479.957 µs]	87.066 µs (23.5%)
iast_TELEMETRY_OFF	471.85 µs [450.652 µs, 493.048 µs]	101.47 µs (27.4%)
tracing	442.305 µs [421.791 µs, 462.819 µs]	71.925 µs (19.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	364.334 µs [344.24 µs, 384.428 µs]	-
iast	480.078 µs [458.986 µs, 501.169 µs]	115.744 µs (31.8%)
iast_FULL	548.575 µs [526.586 µs, 570.564 µs]	184.241 µs (50.6%)
iast_GLOBAL	505.706 µs [483.569 µs, 527.842 µs]	141.372 µs (38.8%)
iast_HARDCODED_SECRET_DISABLED	483.121 µs [461.212 µs, 505.029 µs]	118.787 µs (32.6%)
iast_INACTIVE	454.87 µs [432.95 µs, 476.79 µs]	90.536 µs (24.8%)
iast_TELEMETRY_OFF	466.739 µs [445.661 µs, 487.816 µs]	102.405 µs (28.1%)
tracing	444.594 µs [423.565 µs, 465.622 µs]	80.26 µs (22.0%)

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section baseline
no_agent (1.357 ms) : 1338, 1377
. : milestone, 1357,
appsec (1.722 ms) : 1699, 1746
. : milestone, 1722,
appsec_no_iast (1.737 ms) : 1714, 1761
. : milestone, 1737,
iast (1.456 ms) : 1433, 1479
. : milestone, 1456,
profiling (1.474 ms) : 1450, 1499
. : milestone, 1474,
tracing (1.463 ms) : 1438, 1489
. : milestone, 1463,
section candidate
no_agent (1.361 ms) : 1342, 1380
. : milestone, 1361,
appsec (1.719 ms) : 1694, 1744
. : milestone, 1719,
appsec_no_iast (1.708 ms) : 1684, 1732
. : milestone, 1708,
iast (1.489 ms) : 1466, 1511
. : milestone, 1489,
profiling (1.483 ms) : 1458, 1508
. : milestone, 1483,
tracing (1.457 ms) : 1432, 1482
. : milestone, 1457,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.357 ms [1.338 ms, 1.377 ms]	-
appsec	1.722 ms [1.699 ms, 1.746 ms]	364.794 µs (26.9%)
appsec_no_iast	1.737 ms [1.714 ms, 1.761 ms]	380.039 µs (28.0%)
iast	1.456 ms [1.433 ms, 1.479 ms]	98.32 µs (7.2%)
profiling	1.474 ms [1.45 ms, 1.499 ms]	116.793 µs (8.6%)
tracing	1.463 ms [1.438 ms, 1.489 ms]	105.986 µs (7.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.361 ms [1.342 ms, 1.38 ms]	-
appsec	1.719 ms [1.694 ms, 1.744 ms]	357.731 µs (26.3%)
appsec_no_iast	1.708 ms [1.684 ms, 1.732 ms]	346.151 µs (25.4%)
iast	1.489 ms [1.466 ms, 1.511 ms]	127.203 µs (9.3%)
profiling	1.483 ms [1.458 ms, 1.508 ms]	121.833 µs (8.9%)
tracing	1.457 ms [1.432 ms, 1.482 ms]	95.719 µs (7.0%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/fix-ci-test-selection
git_commit_date	1724059733	1724064974
git_commit_sha	`5ae921b`	`e2947e2`
release_version	1.39.0-SNAPSHOT~5ae921b9c4	1.39.0-SNAPSHOT~e2947e25fc

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1724066752	1724066752
ci_job_id	608592925	608592925
ci_pipeline_id	42101233	42101233
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section baseline
no_agent (15.546 s) : 15546000, 15546000
. : milestone, 15546000,
appsec (14.947 s) : 14947000, 14947000
. : milestone, 14947000,
iast (18.508 s) : 18508000, 18508000
. : milestone, 18508000,
iast_GLOBAL (18.08 s) : 18080000, 18080000
. : milestone, 18080000,
profiling (15.308 s) : 15308000, 15308000
. : milestone, 15308000,
tracing (15.103 s) : 15103000, 15103000
. : milestone, 15103000,
section candidate
no_agent (15.536 s) : 15536000, 15536000
. : milestone, 15536000,
appsec (15.023 s) : 15023000, 15023000
. : milestone, 15023000,
iast (18.894 s) : 18894000, 18894000
. : milestone, 18894000,
iast_GLOBAL (17.983 s) : 17983000, 17983000
. : milestone, 17983000,
profiling (15.132 s) : 15132000, 15132000
. : milestone, 15132000,
tracing (15.022 s) : 15022000, 15022000
. : milestone, 15022000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.546 s [15.546 s, 15.546 s]	-
appsec	14.947 s [14.947 s, 14.947 s]	-599.0 ms (-3.9%)
iast	18.508 s [18.508 s, 18.508 s]	2.962 s (19.1%)
iast_GLOBAL	18.08 s [18.08 s, 18.08 s]	2.534 s (16.3%)
profiling	15.308 s [15.308 s, 15.308 s]	-238.0 ms (-1.5%)
tracing	15.103 s [15.103 s, 15.103 s]	-443.0 ms (-2.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.536 s [15.536 s, 15.536 s]	-
appsec	15.023 s [15.023 s, 15.023 s]	-513.0 ms (-3.3%)
iast	18.894 s [18.894 s, 18.894 s]	3.358 s (21.6%)
iast_GLOBAL	17.983 s [17.983 s, 17.983 s]	2.447 s (15.8%)
profiling	15.132 s [15.132 s, 15.132 s]	-404.0 ms (-2.6%)
tracing	15.022 s [15.022 s, 15.022 s]	-514.0 ms (-3.3%)

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~e2947e25fc, baseline=1.39.0-SNAPSHOT~5ae921b9c4
dateFormat X
axisFormat %s
section baseline
no_agent (1.453 ms) : 1442, 1464
. : milestone, 1453,
appsec (2.216 ms) : 2180, 2252
. : milestone, 2216,
iast (1.955 ms) : 1913, 1997
. : milestone, 1955,
iast_GLOBAL (2.004 ms) : 1961, 2047
. : milestone, 2004,
profiling (1.854 ms) : 1820, 1889
. : milestone, 1854,
tracing (1.839 ms) : 1806, 1872
. : milestone, 1839,
section candidate
no_agent (1.457 ms) : 1445, 1468
. : milestone, 1457,
appsec (2.214 ms) : 2179, 2249
. : milestone, 2214,
iast (1.964 ms) : 1922, 2006
. : milestone, 1964,
iast_GLOBAL (2.002 ms) : 1959, 2045
. : milestone, 2002,
profiling (1.849 ms) : 1815, 1883
. : milestone, 1849,
tracing (1.836 ms) : 1803, 1869
. : milestone, 1836,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.453 ms [1.442 ms, 1.464 ms]	-
appsec	2.216 ms [2.18 ms, 2.252 ms]	763.003 µs (52.5%)
iast	1.955 ms [1.913 ms, 1.997 ms]	502.026 µs (34.6%)
iast_GLOBAL	2.004 ms [1.961 ms, 2.047 ms]	551.136 µs (37.9%)
profiling	1.854 ms [1.82 ms, 1.889 ms]	401.498 µs (27.6%)
tracing	1.839 ms [1.806 ms, 1.872 ms]	385.668 µs (26.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.457 ms [1.445 ms, 1.468 ms]	-
appsec	2.214 ms [2.179 ms, 2.249 ms]	756.987 µs (52.0%)
iast	1.964 ms [1.922 ms, 2.006 ms]	507.524 µs (34.8%)
iast_GLOBAL	2.002 ms [1.959 ms, 2.045 ms]	545.438 µs (37.4%)
profiling	1.849 ms [1.815 ms, 1.883 ms]	392.228 µs (26.9%)
tracing	1.836 ms [1.803 ms, 1.869 ms]	379.562 µs (26.1%)

PerfectSlayer

Thank for taking care of your CI 🤖❤️

The way we called jacoco tasks for gradle will trigger tests outside those selected by `:baseTests` at al. Remove explicit calls to these tasks from CI, and use a `-PcheckCoverage` property instead. When combined with `:baseTests` et al., it will run jacoco tasks only for the selected tests. Remove legacy properties `-PskipInstTests`, `-PskipSmokeTests`, etc. These are now obsolete, and together with the above fix, they do not serve any purpose in CI anymore.

smola added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Aug 19, 2024

smola requested review from a team as code owners August 19, 2024 06:53

smola requested review from a team, PerfectSlayer, dougqh and shatzi and removed request for a teamAugust 19, 2024 06:53

PerfectSlayer reviewed Aug 19, 2024
View reviewed changes

Comment threadbuild.gradle Outdated

smola force-pushed the smola/fix-ci-test-selection branch from 899a5b3 to cda55d5Compare August 19, 2024 07:11

smola requested a review from PerfectSlayer August 19, 2024 07:11

PerfectSlayer approved these changes Aug 19, 2024
View reviewed changes

jpbempel reviewed Aug 19, 2024
View reviewed changes

Comment thread.circleci/config.continue.yml.j2 Outdated

smola force-pushed the smola/fix-ci-test-selection branch from cda55d5 to e2947e2Compare August 19, 2024 10:56

jpbempel approved these changes Aug 19, 2024
View reviewed changes

smola merged commit f21a045 into masterAug 19, 2024

smola deleted the smola/fix-ci-test-selection branch August 19, 2024 13:33

github-actionsBot added this to the 1.39.0 milestone Aug 19, 2024

bric3 mentioned this pull request Jun 29, 2026
Enable coverage when testJvm same as Gradle Daemon JDK or with checkCoverage property #11764
Merged

Navigation Menu

Uh oh!

Fix test selection logic with coverage - #7467

Fix test selection logic with coverage#7467
smola merged 1 commit into
masterfrom
smola/fix-ci-test-selection

smola commented Aug 19, 2024•
edited
Loading

Uh oh!

Uh oh!

pr-commenterBot commented Aug 19, 2024•
edited
Loading

Uh oh!

pr-commenterBot commented Aug 19, 2024•
edited
Loading

Uh oh!

PerfectSlayer left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

smola commented Aug 19, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

Uh oh!

pr-commenterBot commented Aug 19, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

pr-commenterBot commented Aug 19, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smola commented Aug 19, 2024•
edited
Loading

pr-commenterBot commented Aug 19, 2024•
edited
Loading

pr-commenterBot commented Aug 19, 2024•
edited
Loading