Uh oh!
There was an error while loading. Please reload this page.
Fix AWS SDK timeout test flake - #11349
Draft
jordan-wong wants to merge 1 commit into
Draft
Conversation
I can only run on private repositories. |
Contributor
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
jordan-wong
marked this pull request as draft
May 12, 2026 12:18
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:8c720e674d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+391
to
+393
| errorTags AmazonClientException | ||
| } catch (AssertionError e) { | ||
| errorTags SdkClientException, "Unable to execute HTTP request: Request did not complete before the request timeout configuration." | ||
| errorTags IOException |
There was a problem hiding this comment.
For this errored AWS span the tracer still emits error.message, but the class-only errorTags(...) overload only verifies error.type and error.stack; TagsAssert.assertTagsAllVerified() then treats error.message as an unexpected/unverified tag and the test fails whenever either branch matches. The old assertion verified the message, so this flake fix needs to keep asserting error.message flexibly (for example with String or a broad pattern) rather than omitting it.
Useful? React with 👍 / 👎.
ContributorAuthor
There was a problem hiding this comment.
Addressed in the current commit — both branches now pass String as the second argument to errorTags (e.g. errorTags AmazonClientException, String), which asserts that error.message is present and is a String instance without pinning the exact text. This was caught locally too: running the original bot commit against the test produced exactly this failure (set.isEmpty() false, leftover ['error.message']), which is what led to the String matcher approach.
BenchmarksStartupParameters
See matching parameters
SummaryFound 5 performance improvements and 15 performance regressions! Performance is the same for 35 metrics, 10 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.071 s) : 0, 1071407
Total [baseline] (8.856 s) : 0, 8856300
Agent [candidate] (1.097 s) : 0, 1096511
Total [candidate] (8.834 s) : 0, 8833542
section iast
Agent [baseline] (1.258 s) : 0, 1258038
Total [baseline] (9.578 s) : 0, 9577900
Agent [candidate] (1.222 s) : 0, 1222351
Total [candidate] (9.375 s) : 0, 9374668
gantt
title insecure-bank - break down per module: candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.236 ms) : 0, 1236
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (640.383 ms) : 0, 640383
BytebuddyAgent [candidate] (663.01 ms) : 0, 663010
GlobalTracer [baseline] (248.631 ms) : 0, 248631
GlobalTracer [candidate] (281.867 ms) : 0, 281867
AppSec [baseline] (32.874 ms) : 0, 32874
AppSec [candidate] (33.167 ms) : 0, 33167
Debugger [baseline] (60.866 ms) : 0, 60866
Debugger [candidate] (67.01 ms) : 0, 67010
Remote Config [baseline] (606.568 µs) : 0, 607
Remote Config [candidate] (634.756 µs) : 0, 635
Telemetry [baseline] (8.52 ms) : 0, 8520
Telemetry [candidate] (9.082 ms) : 0, 9082
Flare Poller [baseline] (11.56 ms) : 0, 11560
Flare Poller [candidate] (4.71 ms) : 0, 4710
AgentMeter [baseline] (29.703 ms) : 0, 29703
section iast
crashtracking [baseline] (1.23 ms) : 0, 1230
crashtracking [candidate] (1.181 ms) : 0, 1181
BytebuddyAgent [baseline] (833.803 ms) : 0, 833803
BytebuddyAgent [candidate] (795.197 ms) : 0, 795197
GlobalTracer [baseline] (237.793 ms) : 0, 237793
GlobalTracer [candidate] (251.47 ms) : 0, 251470
AppSec [baseline] (31.608 ms) : 0, 31608
AppSec [candidate] (35.312 ms) : 0, 35312
Debugger [baseline] (64.668 ms) : 0, 64668
Debugger [candidate] (63.678 ms) : 0, 63678
Remote Config [baseline] (556.524 µs) : 0, 557
Remote Config [candidate] (612.601 µs) : 0, 613
Telemetry [baseline] (8.119 ms) : 0, 8119
Telemetry [candidate] (8.487 ms) : 0, 8487
Flare Poller [baseline] (3.445 ms) : 0, 3445
Flare Poller [candidate] (3.658 ms) : 0, 3658
AgentMeter [baseline] (11.528 ms) : 0, 11528
IAST [baseline] (28.366 ms) : 0, 28366
IAST [candidate] (27.141 ms) : 0, 27141
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.069 s) : 0, 1069275
Total [baseline] (11.149 s) : 0, 11148843
Agent [candidate] (1.09 s) : 0, 1089956
Total [candidate] (10.859 s) : 0, 10858656
section appsec
Agent [baseline] (1.272 s) : 0, 1271603
Total [baseline] (11.096 s) : 0, 11095892
Agent [candidate] (1.266 s) : 0, 1265833
Total [candidate] (11.119 s) : 0, 11119146
section iast
Agent [baseline] (1.251 s) : 0, 1250990
Total [baseline] (11.151 s) : 0, 11151114
Agent [candidate] (1.235 s) : 0, 1235025
Total [candidate] (11.258 s) : 0, 11257919
section profiling
Agent [baseline] (1.32 s) : 0, 1319716
Total [baseline] (11.173 s) : 0, 11173414
Agent [candidate] (1.217 s) : 0, 1216910
Total [candidate] (10.993 s) : 0, 10993388
gantt
title petclinic - break down per module: candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.233 ms) : 0, 1233
crashtracking [candidate] (1.195 ms) : 0, 1195
BytebuddyAgent [baseline] (636.952 ms) : 0, 636952
BytebuddyAgent [candidate] (659.926 ms) : 0, 659926
GlobalTracer [baseline] (248.983 ms) : 0, 248983
GlobalTracer [candidate] (278.516 ms) : 0, 278516
AppSec [baseline] (32.746 ms) : 0, 32746
AppSec [candidate] (32.694 ms) : 0, 32694
Debugger [baseline] (62.704 ms) : 0, 62704
Debugger [candidate] (68.27 ms) : 0, 68270
Remote Config [baseline] (609.387 µs) : 0, 609
Remote Config [candidate] (646.601 µs) : 0, 647
Telemetry [baseline] (8.642 ms) : 0, 8642
Telemetry [candidate] (8.91 ms) : 0, 8910
Flare Poller [baseline] (10.775 ms) : 0, 10775
Flare Poller [candidate] (3.819 ms) : 0, 3819
AgentMeter [baseline] (29.822 ms) : 0, 29822
section appsec
crashtracking [baseline] (1.243 ms) : 0, 1243
crashtracking [candidate] (1.194 ms) : 0, 1194
BytebuddyAgent [baseline] (679.758 ms) : 0, 679758
BytebuddyAgent [candidate] (694.755 ms) : 0, 694755
GlobalTracer [baseline] (248.767 ms) : 0, 248767
GlobalTracer [candidate] (255.019 ms) : 0, 255019
AppSec [baseline] (185.39 ms) : 0, 185390
AppSec [candidate] (171.205 ms) : 0, 171205
Debugger [baseline] (65.415 ms) : 0, 65415
Debugger [candidate] (69.527 ms) : 0, 69527
Remote Config [baseline] (593.483 µs) : 0, 593
Remote Config [candidate] (793.029 µs) : 0, 793
Telemetry [baseline] (7.698 ms) : 0, 7698
Telemetry [candidate] (9.287 ms) : 0, 9287
Flare Poller [baseline] (9.082 ms) : 0, 9082
Flare Poller [candidate] (3.813 ms) : 0, 3813
AgentMeter [baseline] (12.081 ms) : 0, 12081
IAST [baseline] (24.726 ms) : 0, 24726
IAST [candidate] (24.561 ms) : 0, 24561
section iast
crashtracking [baseline] (1.231 ms) : 0, 1231
crashtracking [candidate] (1.205 ms) : 0, 1205
BytebuddyAgent [baseline] (827.332 ms) : 0, 827332
BytebuddyAgent [candidate] (803.707 ms) : 0, 803707
GlobalTracer [baseline] (237.26 ms) : 0, 237260
GlobalTracer [candidate] (253.788 ms) : 0, 253788
AppSec [baseline] (31.659 ms) : 0, 31659
AppSec [candidate] (34.677 ms) : 0, 34677
Debugger [baseline] (65.039 ms) : 0, 65039
Debugger [candidate] (65.729 ms) : 0, 65729
Remote Config [baseline] (541.96 µs) : 0, 542
Remote Config [candidate] (613.873 µs) : 0, 614
Telemetry [baseline] (7.975 ms) : 0, 7975
Telemetry [candidate] (8.492 ms) : 0, 8492
Flare Poller [baseline] (3.476 ms) : 0, 3476
Flare Poller [candidate] (3.645 ms) : 0, 3645
AgentMeter [baseline] (11.42 ms) : 0, 11420
IAST [baseline] (28.247 ms) : 0, 28247
IAST [candidate] (27.395 ms) : 0, 27395
section profiling
crashtracking [baseline] (528.085 µs) : 0, 528
crashtracking [candidate] (1.227 ms) : 0, 1227
BytebuddyAgent [baseline] (694.758 ms) : 0, 694758
BytebuddyAgent [candidate] (715.057 ms) : 0, 715057
GlobalTracer [baseline] (210.554 ms) : 0, 210554
GlobalTracer [candidate] (217.594 ms) : 0, 217594
AppSec [baseline] (32.64 ms) : 0, 32640
AppSec [candidate] (32.658 ms) : 0, 32658
Debugger [baseline] (68.097 ms) : 0, 68097
Debugger [candidate] (68.76 ms) : 0, 68760
Remote Config [baseline] (582.1 µs) : 0, 582
Remote Config [candidate] (659.11 µs) : 0, 659
Telemetry [baseline] (8.201 ms) : 0, 8201
Telemetry [candidate] (8.946 ms) : 0, 8946
Flare Poller [baseline] (3.71 ms) : 0, 3710
Flare Poller [candidate] (3.85 ms) : 0, 3850
AgentMeter [baseline] (9.475 ms) : 0, 9475
ProfilingAgent [baseline] (93.664 ms) : 0, 93664
ProfilingAgent [candidate] (97.439 ms) : 0, 97439
Profiling [baseline] (94.247 ms) : 0, 94247
Profiling [candidate] (98.022 ms) : 0, 98022
LoadParameters
See matching parameters
SummaryFound 4 performance improvements and 3 performance regressions! Performance is the same for 13 metrics, 16 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section baseline
no_agent (1.273 ms) : 1260, 1285
. : milestone, 1273,
iast (3.376 ms) : 3328, 3423
. : milestone, 3376,
iast_FULL (6.281 ms) : 6216, 6346
. : milestone, 6281,
iast_GLOBAL (3.694 ms) : 3633, 3755
. : milestone, 3694,
profiling (2.354 ms) : 2331, 2377
. : milestone, 2354,
tracing (1.866 ms) : 1852, 1881
. : milestone, 1866,
section candidate
no_agent (1.257 ms) : 1245, 1269
. : milestone, 1257,
iast (3.203 ms) : 3163, 3244
. : milestone, 3203,
iast_FULL (5.825 ms) : 5767, 5883
. : milestone, 5825,
iast_GLOBAL (3.587 ms) : 3532, 3643
. : milestone, 3587,
profiling (2.05 ms) : 2032, 2067
. : milestone, 2050,
tracing (1.962 ms) : 1944, 1979
. : milestone, 1962,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section baseline
no_agent (18.595 ms) : 18402, 18789
. : milestone, 18595,
appsec (18.914 ms) : 18724, 19103
. : milestone, 18914,
code_origins (18.034 ms) : 17856, 18211
. : milestone, 18034,
iast (17.897 ms) : 17720, 18074
. : milestone, 17897,
profiling (18.176 ms) : 17998, 18353
. : milestone, 18176,
tracing (17.894 ms) : 17715, 18072
. : milestone, 17894,
section candidate
no_agent (17.471 ms) : 17293, 17649
. : milestone, 17471,
appsec (18.899 ms) : 18709, 19088
. : milestone, 18899,
code_origins (18.994 ms) : 18804, 19183
. : milestone, 18994,
iast (17.963 ms) : 17782, 18144
. : milestone, 17963,
profiling (18.901 ms) : 18710, 19092
. : milestone, 18901,
tracing (17.869 ms) : 17690, 18047
. : milestone, 17869,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section baseline
no_agent (1.498 ms) : 1486, 1510
. : milestone, 1498,
appsec (3.858 ms) : 3634, 4082
. : milestone, 3858,
iast (2.301 ms) : 2231, 2371
. : milestone, 2301,
iast_GLOBAL (2.345 ms) : 2274, 2415
. : milestone, 2345,
profiling (2.12 ms) : 2065, 2175
. : milestone, 2120,
tracing (2.112 ms) : 2058, 2166
. : milestone, 2112,
section candidate
no_agent (1.501 ms) : 1489, 1512
. : milestone, 1501,
appsec (2.509 ms) : 2456, 2561
. : milestone, 2509,
iast (2.254 ms) : 2188, 2319
. : milestone, 2254,
iast_GLOBAL (2.295 ms) : 2229, 2361
. : milestone, 2295,
profiling (2.125 ms) : 2070, 2179
. : milestone, 2125,
tracing (2.087 ms) : 2035, 2138
. : milestone, 2087,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~8c720e674d, baseline=1.63.0-SNAPSHOT~710ca87583
dateFormat X
axisFormat %s
section baseline
no_agent (15.334 s) : 15334000, 15334000
. : milestone, 15334000,
appsec (14.88 s) : 14880000, 14880000
. : milestone, 14880000,
iast (18.352 s) : 18352000, 18352000
. : milestone, 18352000,
iast_GLOBAL (18.25 s) : 18250000, 18250000
. : milestone, 18250000,
profiling (14.918 s) : 14918000, 14918000
. : milestone, 14918000,
tracing (15.31 s) : 15310000, 15310000
. : milestone, 15310000,
section candidate
no_agent (15.377 s) : 15377000, 15377000
. : milestone, 15377000,
appsec (15.056 s) : 15056000, 15056000
. : milestone, 15056000,
iast (17.999 s) : 17999000, 17999000
. : milestone, 17999000,
iast_GLOBAL (17.656 s) : 17656000, 17656000
. : milestone, 17656000,
profiling (14.949 s) : 14949000, 14949000
. : milestone, 14949000,
tracing (14.658 s) : 14658000, 14658000
. : milestone, 14658000,
|
jordan-wongforce-pushed
the
dd/fix/aws-sdk-timeout-test-flake
branch
2 times, most recently
from
May 20, 2026 14:40
b98381b to
7ae44f7Compare
This comment has been minimized.
This comment has been minimized.
The "timeout and retry errors captured" test could record the AWS timeout as several different exception types depending on where the timeout was enforced (SDK wrapper vs underlying IO abort), and with slightly different message text. The previous try/catch assertion expected one specific (class, message) combination in each branch and intermittently failed when neither matched. Accept either an AmazonClientException (covers SdkClientException and friends) or an IOException, with any String error.message. This matches what the span actually records across the observed flake variants without depending on a stable message string. Remove the @flaky annotation since the assertion is now deterministic under the observed failure modes. Test-only change; no instrumentation behavior modified. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jordan-wongforce-pushed
the
dd/fix/aws-sdk-timeout-test-flake
branch
from
June 1, 2026 17:17
7ae44f7 to
0aafdb1CompareContributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Fixing
timeout and retry errors captured• View in Test Optimization • Questions? Ask in #code-gen-flaky-testsMakes the Spock test
timeout and retry errors captureddeterministic by asserting that the AWS span records an error as either an AWS client exception (AmazonClientExceptionand subclasses) or the underlying networkIOExceptionfamily, without relying on a specific error message string.Motivation
The test was flaky because AWS request timeouts/retries can surface as different exception types depending on where the timeout is enforced (SDK wrapper vs underlying IO abort), and error messages are not stable enough to assert exactly. The previous assertion required a specific exception/message combination, which intermittently didn’t match what was recorded on the span.
Additional Notes
Condition failed with Exception: assertTraces(1) { trace(1) { span { ... } } }Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
PR by Bits - View session in Datadog
Comment @DataDog to request changes