Remove muzzle from CircleCI by randomanderson · Pull Request #8703 · DataDog/dd-trace-java

randomanderson · 2025-04-16T16:56:56Z

What Does This Do

Removes muzzle jobs from CircleCI

Motivation

Eventually, all jobs will be migrated from CircleCI to GitLab. Muzzle jobs in Gitlab have comparable failure rates of 1-3% with a flakyness of 0%. The flakyness indicates that these failures were actual failures and not issues with the CI system.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-04-16T17:40:56Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	landerson/remove-circleci-muzzle
git_commit_date	1746692171	1746715530
git_commit_sha	`d5d53cd`	`5957d2a`
release_version	1.50.0-SNAPSHOT~d5d53cd553	1.50.0-SNAPSHOT~5957d2a141

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746718961	1746718961
ci_job_id	929935167	929935167
ci_pipeline_id	64537113	64537113
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-ps6td5ta-project-304-concurrent-0-lmi7uhur 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-ps6td5ta-project-304-concurrent-0-lmi7uhur 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 56 metrics, 15 unstable metrics.

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.027 s) : 0, 1026873
Total [baseline] (10.509 s) : 0, 10508623
Agent [candidate] (1.022 s) : 0, 1021744
Total [candidate] (10.534 s) : 0, 10534101
section appsec
Agent [baseline] (1.167 s) : 0, 1167269
Total [baseline] (10.815 s) : 0, 10814876
Agent [candidate] (1.161 s) : 0, 1161219
Total [candidate] (10.715 s) : 0, 10714618
section iast
Agent [baseline] (1.156 s) : 0, 1156329
Total [baseline] (10.875 s) : 0, 10874877
Agent [candidate] (1.157 s) : 0, 1156501
Total [candidate] (10.908 s) : 0, 10907541
section profiling
Agent [baseline] (1.281 s) : 0, 1281301
Total [baseline] (10.857 s) : 0, 10857315
Agent [candidate] (1.299 s) : 0, 1299291
Total [candidate] (10.869 s) : 0, 10868849

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.027 s	-
Agent	appsec	1.167 s	140.397 ms (13.7%)
Agent	iast	1.156 s	129.457 ms (12.6%)
Agent	profiling	1.281 s	254.428 ms (24.8%)
Total	tracing	10.509 s	-
Total	appsec	10.815 s	306.253 ms (2.9%)
Total	iast	10.875 s	366.254 ms (3.5%)
Total	profiling	10.857 s	348.692 ms (3.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.022 s	-
Agent	appsec	1.161 s	139.475 ms (13.7%)
Agent	iast	1.157 s	134.757 ms (13.2%)
Agent	profiling	1.299 s	277.547 ms (27.2%)
Total	tracing	10.534 s	-
Total	appsec	10.715 s	180.517 ms (1.7%)
Total	iast	10.908 s	373.44 ms (3.5%)
Total	profiling	10.869 s	334.748 ms (3.2%)

gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (688.791 ms) : 0, 688791
BytebuddyAgent [candidate] (682.86 ms) : 0, 682860
GlobalTracer [baseline] (241.782 ms) : 0, 241782
GlobalTracer [candidate] (239.932 ms) : 0, 239932
AppSec [baseline] (55.033 ms) : 0, 55033
AppSec [candidate] (55.318 ms) : 0, 55318
Debugger [baseline] (7.74 ms) : 0, 7740
Debugger [candidate] (10.434 ms) : 0, 10434
Remote Config [baseline] (705.669 µs) : 0, 706
Remote Config [candidate] (677.076 µs) : 0, 677
Telemetry [baseline] (9.194 ms) : 0, 9194
Telemetry [candidate] (9.114 ms) : 0, 9114
section appsec
BytebuddyAgent [baseline] (705.134 ms) : 0, 705134
BytebuddyAgent [candidate] (700.748 ms) : 0, 700748
GlobalTracer [baseline] (237.461 ms) : 0, 237461
GlobalTracer [candidate] (236.486 ms) : 0, 236486
AppSec [baseline] (175.75 ms) : 0, 175750
AppSec [candidate] (175.618 ms) : 0, 175618
Debugger [baseline] (6.006 ms) : 0, 6006
Debugger [candidate] (5.972 ms) : 0, 5972
Remote Config [baseline] (643.955 µs) : 0, 644
Remote Config [candidate] (623.999 µs) : 0, 624
Telemetry [baseline] (7.84 ms) : 0, 7840
Telemetry [candidate] (7.453 ms) : 0, 7453
IAST [baseline] (21.684 ms) : 0, 21684
IAST [candidate] (21.708 ms) : 0, 21708
section iast
BytebuddyAgent [baseline] (807.993 ms) : 0, 807993
BytebuddyAgent [candidate] (808.503 ms) : 0, 808503
GlobalTracer [baseline] (231.389 ms) : 0, 231389
GlobalTracer [candidate] (231.733 ms) : 0, 231733
AppSec [baseline] (51.925 ms) : 0, 51925
AppSec [candidate] (50.86 ms) : 0, 50860
Debugger [baseline] (5.951 ms) : 0, 5951
Debugger [candidate] (5.868 ms) : 0, 5868
Remote Config [baseline] (597.752 µs) : 0, 598
Remote Config [candidate] (587.39 µs) : 0, 587
Telemetry [baseline] (7.868 ms) : 0, 7868
Telemetry [candidate] (7.941 ms) : 0, 7941
IAST [baseline] (25.398 ms) : 0, 25398
IAST [candidate] (26.697 ms) : 0, 26697
section profiling
BytebuddyAgent [baseline] (673.443 ms) : 0, 673443
BytebuddyAgent [candidate] (683.62 ms) : 0, 683620
GlobalTracer [baseline] (378.377 ms) : 0, 378377
GlobalTracer [candidate] (383.617 ms) : 0, 383617
AppSec [baseline] (54.783 ms) : 0, 54783
AppSec [candidate] (54.584 ms) : 0, 54584
Debugger [baseline] (6.142 ms) : 0, 6142
Debugger [candidate] (6.214 ms) : 0, 6214
Remote Config [baseline] (655.501 µs) : 0, 656
Remote Config [candidate] (725.455 µs) : 0, 725
Telemetry [baseline] (8.128 ms) : 0, 8128
Telemetry [candidate] (8.238 ms) : 0, 8238
ProfilingAgent [baseline] (109.271 ms) : 0, 109271
ProfilingAgent [candidate] (111.437 ms) : 0, 111437
Profiling [baseline] (109.297 ms) : 0, 109297
Profiling [candidate] (111.462 ms) : 0, 111462

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044599
Total [baseline] (8.732 s) : 0, 8731574
Agent [candidate] (1.022 s) : 0, 1021836
Total [candidate] (8.648 s) : 0, 8648324
section iast
Agent [baseline] (1.156 s) : 0, 1155965
Total [baseline] (9.264 s) : 0, 9264452
Agent [candidate] (1.147 s) : 0, 1147454
Total [candidate] (9.251 s) : 0, 9250785
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.15 s) : 0, 1149828
Total [baseline] (9.238 s) : 0, 9237635
Agent [candidate] (1.152 s) : 0, 1152123
Total [candidate] (9.257 s) : 0, 9256904
section iast_TELEMETRY_OFF
Agent [baseline] (1.145 s) : 0, 1145448
Total [baseline] (9.32 s) : 0, 9320267
Agent [candidate] (1.16 s) : 0, 1160469
Total [candidate] (9.267 s) : 0, 9266576

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	iast	1.156 s	111.366 ms (10.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.15 s	105.229 ms (10.1%)
Agent	iast_TELEMETRY_OFF	1.145 s	100.848 ms (9.7%)
Total	tracing	8.732 s	-
Total	iast	9.264 s	532.878 ms (6.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.238 s	506.061 ms (5.8%)
Total	iast_TELEMETRY_OFF	9.32 s	588.693 ms (6.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.022 s	-
Agent	iast	1.147 s	125.618 ms (12.3%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.152 s	130.287 ms (12.8%)
Agent	iast_TELEMETRY_OFF	1.16 s	138.633 ms (13.6%)
Total	tracing	8.648 s	-
Total	iast	9.251 s	602.461 ms (7.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.257 s	608.581 ms (7.0%)
Total	iast_TELEMETRY_OFF	9.267 s	618.253 ms (7.1%)

gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (701.042 ms) : 0, 701042
BytebuddyAgent [candidate] (682.16 ms) : 0, 682160
GlobalTracer [baseline] (244.115 ms) : 0, 244115
GlobalTracer [candidate] (239.786 ms) : 0, 239786
AppSec [baseline] (55.578 ms) : 0, 55578
AppSec [candidate] (55.158 ms) : 0, 55158
Debugger [baseline] (9.936 ms) : 0, 9936
Debugger [candidate] (10.776 ms) : 0, 10776
Remote Config [baseline] (708.155 µs) : 0, 708
Remote Config [candidate] (685.892 µs) : 0, 686
Telemetry [baseline] (9.214 ms) : 0, 9214
Telemetry [candidate] (9.81 ms) : 0, 9810
section iast
BytebuddyAgent [baseline] (807.373 ms) : 0, 807373
BytebuddyAgent [candidate] (801.148 ms) : 0, 801148
GlobalTracer [baseline] (231.363 ms) : 0, 231363
GlobalTracer [candidate] (230.023 ms) : 0, 230023
AppSec [baseline] (50.653 ms) : 0, 50653
AppSec [candidate] (51.142 ms) : 0, 51142
Debugger [baseline] (5.962 ms) : 0, 5962
Debugger [candidate] (5.937 ms) : 0, 5937
Remote Config [baseline] (599.505 µs) : 0, 600
Remote Config [candidate] (586.466 µs) : 0, 586
Telemetry [baseline] (7.941 ms) : 0, 7941
Telemetry [candidate] (7.889 ms) : 0, 7889
IAST [baseline] (27.754 ms) : 0, 27754
IAST [candidate] (27.325 ms) : 0, 27325
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.359 ms) : 0, 802359
BytebuddyAgent [candidate] (803.73 ms) : 0, 803730
GlobalTracer [baseline] (230.699 ms) : 0, 230699
GlobalTracer [candidate] (231.174 ms) : 0, 231174
AppSec [baseline] (49.726 ms) : 0, 49726
AppSec [candidate] (48.982 ms) : 0, 48982
Debugger [baseline] (5.94 ms) : 0, 5940
Debugger [candidate] (5.924 ms) : 0, 5924
Remote Config [baseline] (580.318 µs) : 0, 580
Remote Config [candidate] (587.295 µs) : 0, 587
Telemetry [baseline] (7.951 ms) : 0, 7951
Telemetry [candidate] (7.924 ms) : 0, 7924
IAST [baseline] (28.381 ms) : 0, 28381
IAST [candidate] (30.256 ms) : 0, 30256
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (798.595 ms) : 0, 798595
BytebuddyAgent [candidate] (809.369 ms) : 0, 809369
GlobalTracer [baseline] (230.392 ms) : 0, 230392
GlobalTracer [candidate] (233.321 ms) : 0, 233321
AppSec [baseline] (55.318 ms) : 0, 55318
AppSec [candidate] (55.873 ms) : 0, 55873
Debugger [baseline] (5.962 ms) : 0, 5962
Debugger [candidate] (6.153 ms) : 0, 6153
Remote Config [baseline] (591.224 µs) : 0, 591
Remote Config [candidate] (628.569 µs) : 0, 629
Telemetry [baseline] (7.907 ms) : 0, 7907
Telemetry [candidate] (7.908 ms) : 0, 7908
IAST [baseline] (23.156 ms) : 0, 23156
IAST [candidate] (23.531 ms) : 0, 23531

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-05-08T15:14:56	2025-05-08T15:22:40
git_branch	master	landerson/remove-circleci-muzzle
git_commit_date	1746692171	1746715530
git_commit_sha	`d5d53cd`	`5957d2a`
release_version	1.50.0-SNAPSHOT~d5d53cd553	1.50.0-SNAPSHOT~5957d2a141
start_time	2025-05-08T15:14:42	2025-05-08T15:22:26

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746718155	1746718155
ci_job_id	929935170	929935170
ci_pipeline_id	64537113	64537113
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-ps6td5ta-project-304-concurrent-1-njh5o27u 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-ps6td5ta-project-304-concurrent-1-njh5o27u 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics.

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (1.342 ms) : 1322, 1362
. : milestone, 1342,
appsec (1.721 ms) : 1698, 1745
. : milestone, 1721,
appsec_no_iast (1.694 ms) : 1670, 1717
. : milestone, 1694,
code_origins (1.66 ms) : 1633, 1687
. : milestone, 1660,
iast (1.509 ms) : 1485, 1533
. : milestone, 1509,
profiling (1.511 ms) : 1488, 1533
. : milestone, 1511,
tracing (1.508 ms) : 1483, 1533
. : milestone, 1508,
section candidate
no_agent (1.351 ms) : 1332, 1370
. : milestone, 1351,
appsec (1.736 ms) : 1713, 1759
. : milestone, 1736,
appsec_no_iast (1.721 ms) : 1699, 1744
. : milestone, 1721,
code_origins (1.677 ms) : 1650, 1704
. : milestone, 1677,
iast (1.512 ms) : 1488, 1536
. : milestone, 1512,
profiling (1.502 ms) : 1479, 1526
. : milestone, 1502,
tracing (1.478 ms) : 1454, 1502
. : milestone, 1478,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.342 ms [1.322 ms, 1.362 ms]	-
appsec	1.721 ms [1.698 ms, 1.745 ms]	379.605 µs (28.3%)
appsec_no_iast	1.694 ms [1.67 ms, 1.717 ms]	351.687 µs (26.2%)
code_origins	1.66 ms [1.633 ms, 1.687 ms]	318.096 µs (23.7%)
iast	1.509 ms [1.485 ms, 1.533 ms]	167.482 µs (12.5%)
profiling	1.511 ms [1.488 ms, 1.533 ms]	168.917 µs (12.6%)
tracing	1.508 ms [1.483 ms, 1.533 ms]	166.233 µs (12.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.351 ms [1.332 ms, 1.37 ms]	-
appsec	1.736 ms [1.713 ms, 1.759 ms]	385.346 µs (28.5%)
appsec_no_iast	1.721 ms [1.699 ms, 1.744 ms]	370.535 µs (27.4%)
code_origins	1.677 ms [1.65 ms, 1.704 ms]	325.955 µs (24.1%)
iast	1.512 ms [1.488 ms, 1.536 ms]	160.887 µs (11.9%)
profiling	1.502 ms [1.479 ms, 1.526 ms]	151.291 µs (11.2%)
tracing	1.478 ms [1.454 ms, 1.502 ms]	127.034 µs (9.4%)

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (380.448 µs) : 361, 400
. : milestone, 380,
iast (520.947 µs) : 499, 542
. : milestone, 521,
iast_FULL (731.209 µs) : 709, 753
. : milestone, 731,
iast_GLOBAL (558.383 µs) : 537, 580
. : milestone, 558,
iast_HARDCODED_SECRET_DISABLED (525.431 µs) : 503, 548
. : milestone, 525,
iast_INACTIVE (460.434 µs) : 438, 483
. : milestone, 460,
iast_TELEMETRY_OFF (518.258 µs) : 495, 542
. : milestone, 518,
tracing (458.135 µs) : 436, 480
. : milestone, 458,
section candidate
no_agent (386.926 µs) : 367, 407
. : milestone, 387,
iast (513.323 µs) : 492, 535
. : milestone, 513,
iast_FULL (729.843 µs) : 708, 752
. : milestone, 730,
iast_GLOBAL (565.785 µs) : 544, 587
. : milestone, 566,
iast_HARDCODED_SECRET_DISABLED (514.932 µs) : 493, 537
. : milestone, 515,
iast_INACTIVE (462.758 µs) : 441, 485
. : milestone, 463,
iast_TELEMETRY_OFF (504.573 µs) : 481, 528
. : milestone, 505,
tracing (459.549 µs) : 437, 482
. : milestone, 460,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	380.448 µs [360.867 µs, 400.029 µs]	-
iast	520.947 µs [499.396 µs, 542.498 µs]	140.499 µs (36.9%)
iast_FULL	731.209 µs [709.256 µs, 753.163 µs]	350.761 µs (92.2%)
iast_GLOBAL	558.383 µs [536.518 µs, 580.249 µs]	177.935 µs (46.8%)
iast_HARDCODED_SECRET_DISABLED	525.431 µs [502.773 µs, 548.09 µs]	144.983 µs (38.1%)
iast_INACTIVE	460.434 µs [438.243 µs, 482.625 µs]	79.986 µs (21.0%)
iast_TELEMETRY_OFF	518.258 µs [494.856 µs, 541.66 µs]	137.81 µs (36.2%)
tracing	458.135 µs [435.903 µs, 480.367 µs]	77.687 µs (20.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	386.926 µs [367.244 µs, 406.607 µs]	-
iast	513.323 µs [491.777 µs, 534.868 µs]	126.397 µs (32.7%)
iast_FULL	729.843 µs [707.982 µs, 751.704 µs]	342.917 µs (88.6%)
iast_GLOBAL	565.785 µs [544.298 µs, 587.272 µs]	178.859 µs (46.2%)
iast_HARDCODED_SECRET_DISABLED	514.932 µs [493.015 µs, 536.849 µs]	128.006 µs (33.1%)
iast_INACTIVE	462.758 µs [440.591 µs, 484.925 µs]	75.832 µs (19.6%)
iast_TELEMETRY_OFF	504.573 µs [481.416 µs, 527.729 µs]	117.647 µs (30.4%)
tracing	459.549 µs [437.125 µs, 481.972 µs]	72.623 µs (18.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	landerson/remove-circleci-muzzle
git_commit_date	1746692171	1746715530
git_commit_sha	`d5d53cd`	`5957d2a`
release_version	1.50.0-SNAPSHOT~d5d53cd553	1.50.0-SNAPSHOT~5957d2a141

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1746718635	1746718635
ci_job_id	929935172	929935172
ci_pipeline_id	64537113	64537113
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-ps6td5ta-project-304-concurrent-2-y95sxxzo 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-ps6td5ta-project-304-concurrent-2-y95sxxzo 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (1.471 ms) : 1460, 1483
. : milestone, 1471,
appsec (2.409 ms) : 2360, 2459
. : milestone, 2409,
iast (2.187 ms) : 2125, 2249
. : milestone, 2187,
iast_GLOBAL (2.222 ms) : 2160, 2285
. : milestone, 2222,
profiling (2.044 ms) : 1993, 2096
. : milestone, 2044,
tracing (2.01 ms) : 1962, 2059
. : milestone, 2010,
section candidate
no_agent (1.471 ms) : 1459, 1482
. : milestone, 1471,
appsec (2.4 ms) : 2350, 2449
. : milestone, 2400,
iast (2.176 ms) : 2114, 2238
. : milestone, 2176,
iast_GLOBAL (2.224 ms) : 2161, 2286
. : milestone, 2224,
profiling (2.03 ms) : 1980, 2080
. : milestone, 2030,
tracing (1.999 ms) : 1951, 2047
. : milestone, 1999,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.471 ms [1.46 ms, 1.483 ms]	-
appsec	2.409 ms [2.36 ms, 2.459 ms]	937.987 µs (63.8%)
iast	2.187 ms [2.125 ms, 2.249 ms]	715.634 µs (48.6%)
iast_GLOBAL	2.222 ms [2.16 ms, 2.285 ms]	751.005 µs (51.0%)
profiling	2.044 ms [1.993 ms, 2.096 ms]	572.671 µs (38.9%)
tracing	2.01 ms [1.962 ms, 2.059 ms]	538.882 µs (36.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.471 ms [1.459 ms, 1.482 ms]	-
appsec	2.4 ms [2.35 ms, 2.449 ms]	928.896 µs (63.2%)
iast	2.176 ms [2.114 ms, 2.238 ms]	705.598 µs (48.0%)
iast_GLOBAL	2.224 ms [2.161 ms, 2.286 ms]	752.999 µs (51.2%)
profiling	2.03 ms [1.98 ms, 2.08 ms]	559.219 µs (38.0%)
tracing	1.999 ms [1.951 ms, 2.047 ms]	527.918 µs (35.9%)

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (15.449 s) : 15449000, 15449000
. : milestone, 15449000,
appsec (15.189 s) : 15189000, 15189000
. : milestone, 15189000,
iast (18.55 s) : 18550000, 18550000
. : milestone, 18550000,
iast_GLOBAL (18.299 s) : 18299000, 18299000
. : milestone, 18299000,
profiling (15.057 s) : 15057000, 15057000
. : milestone, 15057000,
tracing (15.071 s) : 15071000, 15071000
. : milestone, 15071000,
section candidate
no_agent (14.873 s) : 14873000, 14873000
. : milestone, 14873000,
appsec (14.997 s) : 14997000, 14997000
. : milestone, 14997000,
iast (18.378 s) : 18378000, 18378000
. : milestone, 18378000,
iast_GLOBAL (18.155 s) : 18155000, 18155000
. : milestone, 18155000,
profiling (15.217 s) : 15217000, 15217000
. : milestone, 15217000,
tracing (14.967 s) : 14967000, 14967000
. : milestone, 14967000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.449 s [15.449 s, 15.449 s]	-
appsec	15.189 s [15.189 s, 15.189 s]	-260.0 ms (-1.7%)
iast	18.55 s [18.55 s, 18.55 s]	3.101 s (20.1%)
iast_GLOBAL	18.299 s [18.299 s, 18.299 s]	2.85 s (18.4%)
profiling	15.057 s [15.057 s, 15.057 s]	-392.0 ms (-2.5%)
tracing	15.071 s [15.071 s, 15.071 s]	-378.0 ms (-2.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.873 s [14.873 s, 14.873 s]	-
appsec	14.997 s [14.997 s, 14.997 s]	124.0 ms (0.8%)
iast	18.378 s [18.378 s, 18.378 s]	3.505 s (23.6%)
iast_GLOBAL	18.155 s [18.155 s, 18.155 s]	3.282 s (22.1%)
profiling	15.217 s [15.217 s, 15.217 s]	344.0 ms (2.3%)
tracing	14.967 s [14.967 s, 14.967 s]	94.0 ms (0.6%)

smola

We should add a required job to the GitLab pipeline and set it to required for merge in the GitHub settings.

PerfectSlayer

Would muzzle runs consistently in GitLab without a pro Nexus? (just checking)

randomanderson · 2025-04-22T16:20:07Z

@PerfectSlayer Pro Nexus is more of a nice to have than a necessity. muzzle has been running on all branches for ~2 months now with the major proxy issues ironed out ~1 month ago.

randomanderson · 2025-04-22T16:21:56Z

@smola the required job in gitlab already exists. When all tests are enabled, it might not be possible because of Gitlab's dependency limit. At that point, we can require the entire pipeline passing

PerfectSlayer · 2025-04-25T14:11:05Z

So is muzzle running from GitLab a required GitHub PR check already?

randomanderson · 2025-04-28T01:46:46Z

@PerfectSlayer I don't have the admin role to change required checks. The required Gitlab job includes muzzle, spotless, and the check_* jobs. The checks jobs are less stable than muzzle but not unreasonably so. (They become 100% stable with the Gitlab tests PR that is also open)

sarahchen6

Perhaps someone with an admin role should set muzzle Gitlab jobs (or the required Gitlab job) as required for PRs in the meantime, but otherwise LGTM!

PerfectSlayer · 2025-05-06T05:19:59Z

I don’t think you can do it on the UI. We will need to code some GH API call to update the PR checks requirements.
What about muzzle pipeline stability in GitLab and Maven issues due to the lack of Nexus Repository pro / Magic Mirror support?

bric3 · 2025-05-06T10:04:48Z

Same remarks as @PerfectSlayer

randomanderson · 2025-05-06T23:31:59Z

@PerfectSlayer @bric3 Turns out, muzzle checks weren't using the proxy. During the rate limit incident last week, muzzle failures in Gitlab rose to ~10%. (Fix Here)

The stability will only increase when the proxy is used.

PerfectSlayer · 2025-05-07T15:13:31Z

@PerfectSlayer I don't have the admin role to change required checks. The required Gitlab job includes muzzle, spotless, and the check_* jobs. The checks jobs are less stable than muzzle but not unreasonably so. (They become 100% stable with the Gitlab tests PR that is also open)

We just migrated #8771 so you would be able to add them from there 🙌
Keep in mind the gatekeeper check is not required yet as we are still investigating the stability of the pipelines it requires.

And I will let @bric3 and @sarahchen6 approve, migrate, and ensure removal the CI pipelines to prevent being myself a blocker of the migration :)

sarahchen6 · 2025-05-07T15:28:38Z

We just migrated #8771 so you would be able to add them from there 🙌
Keep in mind the gatekeeper check is not required yet as we are still investigating the stability of the pipelines it requires.

Also, the gatekeeper check currently excludes Gitlab muzzle checks due to low success rate. I'm thinking we should probably wait to merge this PR until after the muzzle checks stabilize, they are removed from the excluded list in the gatekeeper check, and the gatekeeper check is required. 🤔

…rcleci-muzzle

randomanderson · 2025-05-08T15:01:08Z

@sarahchen6 I removed muzzle from the all-green excluded list and think this PR is ready to merge

Looking at the past month for muzzle only,

CircleCI 6.7 % failures
Gitlab 1.8%-6.5% failure (there are 8 jobs)
CircleCI 0% flakiness
Gitlab 0.14%-0.56% flakiness

This is with the Maven Central incident, updates to the internal proxy, and actual muzzle failures because of the release of new libraries.

Muzzle is very stable in Gitlab (less failures than CircleCI and only one-half of one-percent flakiness at worse).

Comparatively, z_test_8_inst has a failure rate of 16% and a flakiness of ~3.8% and is a required job.

sarahchen6 · 2025-05-08T15:32:30Z

My only concern would be that the gatekeeping job is not yet required, so there would be nothing requiring muzzle tests to pass before merging a PR. Otherwise, the numbers look good.

Remove muzzle from CircleCI
12273ce

randomanderson added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Apr 16, 2025

randomanderson requested a review from a team as a code owner April 16, 2025 16:56

randomanderson requested a review from ygree April 16, 2025 16:56

smola reviewed Apr 21, 2025
View reviewed changes

PerfectSlayer reviewed Apr 22, 2025
View reviewed changes

sarahchen6 approved these changes May 5, 2025
View reviewed changes

randomanderson added 2 commits May 8, 2025 10:40

Merge remote-tracking branch 'origin/master' into landerson/remove-ci…
6fdcc81
…rcleci-muzzle

add muzzle as part of all-green
5957d2a

randomanderson enabled auto-merge (squash) May 9, 2025 15:13

randomanderson merged commit c81bba3 into masterMay 9, 2025

randomanderson deleted the landerson/remove-circleci-muzzle branch May 9, 2025 15:37

github-actionsBot added this to the 1.50.0 milestone May 9, 2025

Uh oh!

Conversation

randomanderson commented Apr 16, 2025

What Does This Do

Motivation

Contributor Checklist

Uh oh!

pr-commenterBot commented Apr 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

smola left a comment

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

randomanderson commented Apr 22, 2025

Uh oh!

randomanderson commented Apr 22, 2025

Uh oh!

PerfectSlayer commented Apr 25, 2025

Uh oh!

randomanderson commented Apr 28, 2025

Uh oh!

sarahchen6 left a comment

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer commented May 6, 2025

Uh oh!

bric3 commented May 6, 2025

Uh oh!

randomanderson commented May 6, 2025

Uh oh!

PerfectSlayer commented May 7, 2025

Uh oh!

sarahchen6 commented May 7, 2025

Uh oh!

randomanderson commented May 8, 2025

Uh oh!

sarahchen6 commented May 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pr-commenterBot commented Apr 16, 2025•
edited
Loading