add recipe for handling flaky tests by emmettbutler · Pull Request #9020 · DataDog/dd-trace-java

emmettbutler · 2025-06-20T19:29:49Z

What Does This Do

This change adds some documentation explaining how to handle flaky tests when they occur on pull requests. It also removes a few outdated references to CircleCI.

Motivation

The motivation is to make it easier for new contributors to get up to speed on the project.

Jira ticket: LANGPLAT-352

github-actions · 2025-06-20T19:29:59Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Remove the tag from the pull request title

If you need help, please check our contributing guidelines.

pr-commenter · 2025-06-20T20:03:19Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	emmett.butler/flaky-guidance
git_commit_date	1750427857	1750447681
git_commit_sha	`60c30c2`	`f8ec06c`
release_version	1.51.0-SNAPSHOT~60c30c276e	1.51.0-SNAPSHOT~f8ec06c202

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750449582	1750449582
ci_job_id	992078701	992078701
ci_pipeline_id	68361068	68361068
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-clrnfvyp-project-304-concurrent-0-4979j6gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-clrnfvyp-project-304-concurrent-0-4979j6gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 10 unstable metrics.

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1025897
Total [baseline] (10.504 s) : 0, 10503768
Agent [candidate] (1.025 s) : 0, 1025436
Total [candidate] (10.643 s) : 0, 10642590
section appsec
Agent [baseline] (1.188 s) : 0, 1187607
Total [baseline] (10.833 s) : 0, 10832560
Agent [candidate] (1.185 s) : 0, 1185209
Total [candidate] (10.773 s) : 0, 10772905
section iast
Agent [baseline] (1.151 s) : 0, 1151015
Total [baseline] (10.848 s) : 0, 10847997
Agent [candidate] (1.161 s) : 0, 1160964
Total [candidate] (10.864 s) : 0, 10863621
section profiling
Agent [baseline] (1.268 s) : 0, 1268441
Total [baseline] (10.952 s) : 0, 10951651
Agent [candidate] (1.273 s) : 0, 1272572
Total [candidate] (10.991 s) : 0, 10990852

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.026 s	-
Agent	appsec	1.188 s	161.71 ms (15.8%)
Agent	iast	1.151 s	125.118 ms (12.2%)
Agent	profiling	1.268 s	242.543 ms (23.6%)
Total	tracing	10.504 s	-
Total	appsec	10.833 s	328.792 ms (3.1%)
Total	iast	10.848 s	344.229 ms (3.3%)
Total	profiling	10.952 s	447.883 ms (4.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.025 s	-
Agent	appsec	1.185 s	159.773 ms (15.6%)
Agent	iast	1.161 s	135.528 ms (13.2%)
Agent	profiling	1.273 s	247.136 ms (24.1%)
Total	tracing	10.643 s	-
Total	appsec	10.773 s	130.315 ms (1.2%)
Total	iast	10.864 s	221.031 ms (2.1%)
Total	profiling	10.991 s	348.263 ms (3.3%)

gantt
title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.68 ms) : 0, 686680
BytebuddyAgent [candidate] (685.804 ms) : 0, 685804
GlobalTracer [baseline] (242.344 ms) : 0, 242344
GlobalTracer [candidate] (242.107 ms) : 0, 242107
AppSec [baseline] (61.238 ms) : 0, 61238
AppSec [candidate] (58.359 ms) : 0, 58359
Debugger [baseline] (6.097 ms) : 0, 6097
Debugger [candidate] (7.007 ms) : 0, 7007
Remote Config [baseline] (645.145 µs) : 0, 645
Remote Config [candidate] (669.307 µs) : 0, 669
Telemetry [baseline] (8.003 ms) : 0, 8003
Telemetry [candidate] (10.641 ms) : 0, 10641
section appsec
BytebuddyAgent [baseline] (714.341 ms) : 0, 714341
BytebuddyAgent [candidate] (713.031 ms) : 0, 713031
GlobalTracer [baseline] (236.799 ms) : 0, 236799
GlobalTracer [candidate] (236.308 ms) : 0, 236308
IAST [baseline] (22.015 ms) : 0, 22015
IAST [candidate] (22.129 ms) : 0, 22129
AppSec [baseline] (179.877 ms) : 0, 179877
AppSec [candidate] (179.283 ms) : 0, 179283
Debugger [baseline] (5.809 ms) : 0, 5809
Debugger [candidate] (5.77 ms) : 0, 5770
Remote Config [baseline] (604.759 µs) : 0, 605
Remote Config [candidate] (606.005 µs) : 0, 606
Telemetry [baseline] (7.265 ms) : 0, 7265
Telemetry [candidate] (7.216 ms) : 0, 7216
section iast
BytebuddyAgent [baseline] (804.712 ms) : 0, 804712
BytebuddyAgent [candidate] (812.268 ms) : 0, 812268
GlobalTracer [baseline] (231.61 ms) : 0, 231610
GlobalTracer [candidate] (233.392 ms) : 0, 233392
IAST [baseline] (25.285 ms) : 0, 25285
IAST [candidate] (27.029 ms) : 0, 27029
AppSec [baseline] (54.034 ms) : 0, 54034
AppSec [candidate] (52.633 ms) : 0, 52633
Debugger [baseline] (5.927 ms) : 0, 5927
Debugger [candidate] (5.999 ms) : 0, 5999
Remote Config [baseline] (601.135 µs) : 0, 601
Remote Config [candidate] (629.193 µs) : 0, 629
Telemetry [baseline] (7.972 ms) : 0, 7972
Telemetry [candidate] (8.044 ms) : 0, 8044
section profiling
BytebuddyAgent [baseline] (676.044 ms) : 0, 676044
BytebuddyAgent [candidate] (677.589 ms) : 0, 677589
GlobalTracer [baseline] (360.592 ms) : 0, 360592
GlobalTracer [candidate] (361.396 ms) : 0, 361396
AppSec [baseline] (62.258 ms) : 0, 62258
AppSec [candidate] (62.303 ms) : 0, 62303
Debugger [baseline] (6.09 ms) : 0, 6090
Debugger [candidate] (6.097 ms) : 0, 6097
Remote Config [baseline] (705.341 µs) : 0, 705
Remote Config [candidate] (706.053 µs) : 0, 706
Telemetry [baseline] (8.06 ms) : 0, 8060
Telemetry [candidate] (8.135 ms) : 0, 8135
ProfilingAgent [baseline] (106.325 ms) : 0, 106325
ProfilingAgent [candidate] (107.586 ms) : 0, 107586
Profiling [baseline] (106.35 ms) : 0, 106350
Profiling [candidate] (107.612 ms) : 0, 107612

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.03 s) : 0, 1030421
Total [baseline] (8.552 s) : 0, 8551769
Agent [candidate] (1.025 s) : 0, 1025058
Total [candidate] (8.568 s) : 0, 8567647
section iast
Agent [baseline] (1.148 s) : 0, 1147624
Total [baseline] (9.204 s) : 0, 9203944
Agent [candidate] (1.152 s) : 0, 1152074
Total [candidate] (9.219 s) : 0, 9219466

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.03 s	-
Agent	iast	1.148 s	117.203 ms (11.4%)
Total	tracing	8.552 s	-
Total	iast	9.204 s	652.176 ms (7.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.025 s	-
Agent	iast	1.152 s	127.015 ms (12.4%)
Total	tracing	8.568 s	-
Total	iast	9.219 s	651.819 ms (7.6%)

gantt
title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (691.091 ms) : 0, 691091
BytebuddyAgent [candidate] (685.648 ms) : 0, 685648
GlobalTracer [baseline] (242.007 ms) : 0, 242007
GlobalTracer [candidate] (242.091 ms) : 0, 242091
AppSec [baseline] (55.663 ms) : 0, 55663
AppSec [candidate] (58.542 ms) : 0, 58542
Debugger [baseline] (7.053 ms) : 0, 7053
Debugger [candidate] (7.611 ms) : 0, 7611
Remote Config [baseline] (697.43 µs) : 0, 697
Remote Config [candidate] (657.787 µs) : 0, 658
Telemetry [baseline] (12.886 ms) : 0, 12886
Telemetry [candidate] (9.628 ms) : 0, 9628
section iast
BytebuddyAgent [baseline] (802.759 ms) : 0, 802759
BytebuddyAgent [candidate] (805.568 ms) : 0, 805568
GlobalTracer [baseline] (231.144 ms) : 0, 231144
GlobalTracer [candidate] (232.076 ms) : 0, 232076
IAST [baseline] (27.331 ms) : 0, 27331
IAST [candidate] (26.119 ms) : 0, 26119
AppSec [baseline] (51.178 ms) : 0, 51178
AppSec [candidate] (53.023 ms) : 0, 53023
Debugger [baseline] (5.835 ms) : 0, 5835
Debugger [candidate] (5.932 ms) : 0, 5932
Remote Config [baseline] (602.001 µs) : 0, 602
Remote Config [candidate] (597.377 µs) : 0, 597
Telemetry [baseline] (7.973 ms) : 0, 7973
Telemetry [candidate] (7.96 ms) : 0, 7960

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	emmett.butler/flaky-guidance
git_commit_date	1750427857	1750447681
git_commit_sha	`60c30c2`	`f8ec06c`
release_version	1.51.0-SNAPSHOT~60c30c276e	1.51.0-SNAPSHOT~f8ec06c202

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750449252	1750449252
ci_job_id	992078702	992078702
ci_pipeline_id	68361068	68361068
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-bziamzy-project-304-concurrent-0-kviyhrps 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-bziamzy-project-304-concurrent-0-kviyhrps 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 1 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:iast:high_load	better [-560.820µs; -235.932µs] or [-5.950%; -2.503%]	unstable [-48.791op/s; +91.354op/s] or [-9.906%; +18.547%]	9.027ms	513.844op/s	9.425ms	492.562op/s
scenario:load:insecure-bank:tracing:high_load	better [-605.283µs; -349.509µs] or [-7.449%; -4.301%]	unstable [-46.962op/s; +116.524op/s] or [-8.232%; +20.426%]	7.648ms	605.250op/s	8.125ms	570.469op/s
scenario:load:petclinic:appsec:high_load	worse [+1.484ms; +2.357ms] or [+3.231%; +5.131%]	unstable [-10.528op/s; +2.353op/s] or [-10.339%; +2.311%]	47.869ms	97.737op/s	45.948ms	101.825op/s
scenario:load:petclinic:iast:high_load	better [-2.365ms; -1.548ms] or [-5.258%; -3.443%]	unstable [-2.544op/s; +11.969op/s] or [-2.446%; +11.509%]	43.020ms	108.713op/s	44.977ms	104.000op/s

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (37.529 ms) : 37223, 37835
. : milestone, 37529,
appsec (45.948 ms) : 45558, 46338
. : milestone, 45948,
code_origins (44.91 ms) : 44527, 45293
. : milestone, 44910,
iast (44.977 ms) : 44581, 45373
. : milestone, 44977,
profiling (48.746 ms) : 48309, 49182
. : milestone, 48746,
tracing (42.558 ms) : 42226, 42891
. : milestone, 42558,
section candidate
no_agent (36.583 ms) : 36293, 36873
. : milestone, 36583,
appsec (47.869 ms) : 47448, 48290
. : milestone, 47869,
code_origins (44.016 ms) : 43649, 44383
. : milestone, 44016,
iast (43.02 ms) : 42658, 43382
. : milestone, 43020,
profiling (48.052 ms) : 47584, 48521
. : milestone, 48052,
tracing (42.79 ms) : 42450, 43130
. : milestone, 42790,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.529 ms [37.223 ms, 37.835 ms]	-
appsec	45.948 ms [45.558 ms, 46.338 ms]	8.419 ms (22.4%)
code_origins	44.91 ms [44.527 ms, 45.293 ms]	7.381 ms (19.7%)
iast	44.977 ms [44.581 ms, 45.373 ms]	7.448 ms (19.8%)
profiling	48.746 ms [48.309 ms, 49.182 ms]	11.217 ms (29.9%)
tracing	42.558 ms [42.226 ms, 42.891 ms]	5.03 ms (13.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.583 ms [36.293 ms, 36.873 ms]	-
appsec	47.869 ms [47.448 ms, 48.29 ms]	11.286 ms (30.9%)
code_origins	44.016 ms [43.649 ms, 44.383 ms]	7.433 ms (20.3%)
iast	43.02 ms [42.658 ms, 43.382 ms]	6.437 ms (17.6%)
profiling	48.052 ms [47.584 ms, 48.521 ms]	11.469 ms (31.4%)
tracing	42.79 ms [42.45 ms, 43.13 ms]	6.207 ms (17.0%)

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (4.387 ms) : 4337, 4436
. : milestone, 4387,
iast (9.425 ms) : 9269, 9581
. : milestone, 9425,
iast_FULL (13.818 ms) : 13544, 14091
. : milestone, 13818,
iast_GLOBAL (9.898 ms) : 9726, 10070
. : milestone, 9898,
profiling (8.476 ms) : 8336, 8615
. : milestone, 8476,
tracing (8.125 ms) : 8000, 8251
. : milestone, 8125,
section candidate
no_agent (4.468 ms) : 4414, 4523
. : milestone, 4468,
iast (9.027 ms) : 8880, 9173
. : milestone, 9027,
iast_FULL (14.096 ms) : 13812, 14379
. : milestone, 14096,
iast_GLOBAL (10.207 ms) : 10025, 10389
. : milestone, 10207,
profiling (8.772 ms) : 8632, 8911
. : milestone, 8772,
tracing (7.648 ms) : 7536, 7759
. : milestone, 7648,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.387 ms [4.337 ms, 4.436 ms]	-
iast	9.425 ms [9.269 ms, 9.581 ms]	5.038 ms (114.9%)
iast_FULL	13.818 ms [13.544 ms, 14.091 ms]	9.431 ms (215.0%)
iast_GLOBAL	9.898 ms [9.726 ms, 10.07 ms]	5.511 ms (125.6%)
profiling	8.476 ms [8.336 ms, 8.615 ms]	4.089 ms (93.2%)
tracing	8.125 ms [8.0 ms, 8.251 ms]	3.739 ms (85.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.468 ms [4.414 ms, 4.523 ms]	-
iast	9.027 ms [8.88 ms, 9.173 ms]	4.558 ms (102.0%)
iast_FULL	14.096 ms [13.812 ms, 14.379 ms]	9.628 ms (215.5%)
iast_GLOBAL	10.207 ms [10.025 ms, 10.389 ms]	5.739 ms (128.4%)
profiling	8.772 ms [8.632 ms, 8.911 ms]	4.304 ms (96.3%)
tracing	7.648 ms [7.536 ms, 7.759 ms]	3.18 ms (71.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	emmett.butler/flaky-guidance
git_commit_date	1750427857	1750447681
git_commit_sha	`60c30c2`	`f8ec06c`
release_version	1.51.0-SNAPSHOT~60c30c276e	1.51.0-SNAPSHOT~f8ec06c202

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1750449763	1750449763
ci_job_id	992078703	992078703
ci_pipeline_id	68361068	68361068
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-bziamzy-project-304-concurrent-1-fv58j1gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-bziamzy-project-304-concurrent-1-fv58j1gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (15.59 s) : 15590000, 15590000
. : milestone, 15590000,
appsec (15.101 s) : 15101000, 15101000
. : milestone, 15101000,
iast (18.597 s) : 18597000, 18597000
. : milestone, 18597000,
iast_GLOBAL (18.075 s) : 18075000, 18075000
. : milestone, 18075000,
profiling (15.369 s) : 15369000, 15369000
. : milestone, 15369000,
tracing (15.156 s) : 15156000, 15156000
. : milestone, 15156000,
section candidate
no_agent (14.768 s) : 14768000, 14768000
. : milestone, 14768000,
appsec (14.961 s) : 14961000, 14961000
. : milestone, 14961000,
iast (18.938 s) : 18938000, 18938000
. : milestone, 18938000,
iast_GLOBAL (17.98 s) : 17980000, 17980000
. : milestone, 17980000,
profiling (15.309 s) : 15309000, 15309000
. : milestone, 15309000,
tracing (14.932 s) : 14932000, 14932000
. : milestone, 14932000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.59 s [15.59 s, 15.59 s]	-
appsec	15.101 s [15.101 s, 15.101 s]	-489.0 ms (-3.1%)
iast	18.597 s [18.597 s, 18.597 s]	3.007 s (19.3%)
iast_GLOBAL	18.075 s [18.075 s, 18.075 s]	2.485 s (15.9%)
profiling	15.369 s [15.369 s, 15.369 s]	-221.0 ms (-1.4%)
tracing	15.156 s [15.156 s, 15.156 s]	-434.0 ms (-2.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.768 s [14.768 s, 14.768 s]	-
appsec	14.961 s [14.961 s, 14.961 s]	193.0 ms (1.3%)
iast	18.938 s [18.938 s, 18.938 s]	4.17 s (28.2%)
iast_GLOBAL	17.98 s [17.98 s, 17.98 s]	3.212 s (21.7%)
profiling	15.309 s [15.309 s, 15.309 s]	541.0 ms (3.7%)
tracing	14.932 s [14.932 s, 14.932 s]	164.0 ms (1.1%)

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (1.475 ms) : 1463, 1486
. : milestone, 1475,
appsec (2.396 ms) : 2347, 2444
. : milestone, 2396,
iast (2.194 ms) : 2132, 2255
. : milestone, 2194,
iast_GLOBAL (2.222 ms) : 2160, 2283
. : milestone, 2222,
profiling (2.048 ms) : 1998, 2099
. : milestone, 2048,
tracing (2.009 ms) : 1961, 2056
. : milestone, 2009,
section candidate
no_agent (1.475 ms) : 1464, 1487
. : milestone, 1475,
appsec (2.393 ms) : 2345, 2442
. : milestone, 2393,
iast (2.192 ms) : 2130, 2253
. : milestone, 2192,
iast_GLOBAL (2.227 ms) : 2165, 2289
. : milestone, 2227,
profiling (2.035 ms) : 1986, 2085
. : milestone, 2035,
tracing (1.996 ms) : 1949, 2043
. : milestone, 1996,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.463 ms, 1.486 ms]	-
appsec	2.396 ms [2.347 ms, 2.444 ms]	921.382 µs (62.5%)
iast	2.194 ms [2.132 ms, 2.255 ms]	719.104 µs (48.8%)
iast_GLOBAL	2.222 ms [2.16 ms, 2.283 ms]	747.106 µs (50.7%)
profiling	2.048 ms [1.998 ms, 2.099 ms]	573.892 µs (38.9%)
tracing	2.009 ms [1.961 ms, 2.056 ms]	534.218 µs (36.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.464 ms, 1.487 ms]	-
appsec	2.393 ms [2.345 ms, 2.442 ms]	918.427 µs (62.3%)
iast	2.192 ms [2.13 ms, 2.253 ms]	716.564 µs (48.6%)
iast_GLOBAL	2.227 ms [2.165 ms, 2.289 ms]	751.884 µs (51.0%)
profiling	2.035 ms [1.986 ms, 2.085 ms]	560.378 µs (38.0%)
tracing	1.996 ms [1.949 ms, 2.043 ms]	521.097 µs (35.3%)

PerfectSlayer · 2025-06-23T05:46:51Z

-If you would like to run another version of the system tests on continuous integration, or update them to the latest version, you would need to use [the update pinned system tests script](../.circleci/update_pinned_system_tests.sh) as your pull request won't use the latest `main` version from the system test repository, but a pinned version. 
-


We might need to keep this part but update it. @TonyCTHsu is working on pinning the system tests again.

PerfectSlayer · 2025-06-23T05:50:22Z

-ands information on trace checks being performed.
-Logs can be viewed in CircleCI within the Test-Agent container step for all instrumentation test suites, ie: `z_test_8_inst` job.
+ands information on trace checks being performed.
+Logs can be viewed in CI within the Test-Agent container step for all instrumentation test suites, ie: `z_test_8_inst` job.


Is there still a "Test-Agent container step"?
The only thing I could find was:
APM Test Agent is running. (HTTP 200) All APM Test Agent Check Traces returned successful! (HTTP 200) APM Test Agent Check Traces Summary Results:

I don't know. Maybe we can answer that separately from this merge.

PerfectSlayer · 2025-06-23T05:52:20Z

I also wonder if we can share links to our confluence docs about how to use / leverage our GitLab CI?

emmettbutler · 2025-07-03T17:11:53Z

@PerfectSlayer that's a great idea. I'll open another pull request to add those.

sarahchen6 · 2025-07-03T17:42:28Z

Just realized, but #9088 has a lot of overlapping changes

PerfectSlayer · 2025-08-05T12:11:25Z

Closing PR to preview review reminder.
Feel free to re-open it if needed!

add recipe for handling flaky tests
f8ec06c

emmettbutler requested a review from a team as a code owner June 20, 2025 19:29

emmettbutler requested a review from PerfectSlayer June 20, 2025 19:29

mcculls added type: documentation comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes labels Jun 20, 2025

mcculls changed the title ~~add recipe for handling flaky tests [LANGPLAT-352]~~add recipe for handling flaky testsJun 20, 2025

PerfectSlayer reviewed Jun 23, 2025
View reviewed changes

emmettbutler requested a review from PerfectSlayer July 3, 2025 17:11

PerfectSlayer closed this Aug 5, 2025

Navigation Menu

Uh oh!

add recipe for handling flaky tests - #9020

add recipe for handling flaky tests#9020
emmettbutler wants to merge 1 commit into
masterfrom
emmett.butler/flaky-guidance

emmettbutler commented Jun 20, 2025•
edited by atlassian Bot
Loading

Uh oh!

github-actionsBot commented Jun 20, 2025•
edited
Loading

Uh oh!

pr-commenterBot commented Jun 20, 2025

Uh oh!

PerfectSlayerJun 23, 2025

Uh oh!

PerfectSlayerJun 23, 2025

Uh oh!

emmettbutlerJul 3, 2025

Uh oh!

PerfectSlayer commented Jun 23, 2025

Uh oh!

emmettbutler commented Jul 3, 2025

Uh oh!

sarahchen6 commented Jul 3, 2025

Uh oh!

PerfectSlayer commented Aug 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		If you would like to run another version of the system tests on continuous integration, or update them to the latest version, you would need to use [the update pinned system tests script](../.circleci/update_pinned_system_tests.sh) as your pull request won't use the latest `main` version from the system test repository, but a pinned version.

Uh oh!

Conversation

emmettbutler commented Jun 20, 2025• edited by atlassian BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Uh oh!

github-actionsBot commented Jun 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenterBot commented Jun 20, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

PerfectSlayerJun 23, 2025

Choose a reason for hiding this comment

Uh oh!

PerfectSlayerJun 23, 2025

Choose a reason for hiding this comment

Uh oh!

emmettbutlerJul 3, 2025

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer commented Jun 23, 2025

Uh oh!

emmettbutler commented Jul 3, 2025

Uh oh!

sarahchen6 commented Jul 3, 2025

Uh oh!

PerfectSlayer commented Aug 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

emmettbutler commented Jun 20, 2025•
edited by atlassian Bot
Loading

github-actionsBot commented Jun 20, 2025•
edited
Loading