Uh oh!
There was an error while loading. Please reload this page.
Add mechanism to remove exception probes - #8163
Closed
jpbempel wants to merge 1 commit into
Closed
Conversation
Add circuit breaker when evaluating exception probe to detect when an exception probe is executed in a hot path only waiting for exception to happen. when circuit breaker is open after 1000 calls in 10 seconds window, we check if there is no exception captured during this period too and remove the probe from the configuration and un-instrument it. This will prevent too much overhead for hot path
Debugger benchmarksParameters
See matching parameters
SummaryFound 0 performance improvements and 5 performance regressions! Performance is the same for 3 metrics, 7 unstable metrics.
See unchanged results
Request duration reports for reportsgantt
title reports - request duration [CI 0.99] : candidate=None, baseline=None
dateFormat X
axisFormat %s
section baseline
noprobe (313.955 µs) : 272, 355
. : milestone, 314,
basic (308.553 µs) : 298, 319
. : milestone, 309,
loop (10.589 ms) : 10528, 10649
. : milestone, 10589,
section candidate
noprobe (314.294 µs) : 270, 358
. : milestone, 314,
basic (305.025 µs) : 294, 316
. : milestone, 305,
loop (11.034 ms) : 11002, 11066
. : milestone, 11034,
|
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 4 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058483
Total [baseline] (10.438 s) : 0, 10437503
Agent [candidate] (1.059 s) : 0, 1059063
Total [candidate] (10.481 s) : 0, 10480904
section appsec
Agent [baseline] (1.187 s) : 0, 1186692
Total [baseline] (10.675 s) : 0, 10674633
Agent [candidate] (1.188 s) : 0, 1188008
Total [candidate] (10.696 s) : 0, 10696005
section iast
Agent [baseline] (1.185 s) : 0, 1184787
Total [baseline] (11.034 s) : 0, 11034455
Agent [candidate] (1.195 s) : 0, 1194514
Total [candidate] (11.049 s) : 0, 11048876
section profiling
Agent [baseline] (1.281 s) : 0, 1281133
Total [baseline] (10.796 s) : 0, 10795613
Agent [candidate] (1.289 s) : 0, 1289019
Total [candidate] (10.877 s) : 0, 10877109
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (716.34 ms) : 0, 716340
BytebuddyAgent [candidate] (714.061 ms) : 0, 714061
GlobalTracer [baseline] (255.75 ms) : 0, 255750
GlobalTracer [candidate] (258.126 ms) : 0, 258126
AppSec [baseline] (56.061 ms) : 0, 56061
AppSec [candidate] (55.814 ms) : 0, 55814
Remote Config [baseline] (724.048 µs) : 0, 724
Remote Config [candidate] (744.409 µs) : 0, 744
Telemetry [baseline] (14.578 ms) : 0, 14578
Telemetry [candidate] (15.272 ms) : 0, 15272
section appsec
BytebuddyAgent [baseline] (729.339 ms) : 0, 729339
BytebuddyAgent [candidate] (730.73 ms) : 0, 730730
GlobalTracer [baseline] (253.009 ms) : 0, 253009
GlobalTracer [candidate] (253.366 ms) : 0, 253366
AppSec [baseline] (170.842 ms) : 0, 170842
AppSec [candidate] (170.114 ms) : 0, 170114
Remote Config [baseline] (658.715 µs) : 0, 659
Remote Config [candidate] (657.089 µs) : 0, 657
Telemetry [baseline] (8.171 ms) : 0, 8171
Telemetry [candidate] (8.535 ms) : 0, 8535
IAST [baseline] (19.257 ms) : 0, 19257
IAST [candidate] (19.238 ms) : 0, 19238
section iast
BytebuddyAgent [baseline] (833.626 ms) : 0, 833626
BytebuddyAgent [candidate] (839.293 ms) : 0, 839293
GlobalTracer [baseline] (247.498 ms) : 0, 247498
GlobalTracer [candidate] (250.001 ms) : 0, 250001
AppSec [baseline] (57.924 ms) : 0, 57924
AppSec [candidate] (58.621 ms) : 0, 58621
Remote Config [baseline] (661.379 µs) : 0, 661
Remote Config [candidate] (668.568 µs) : 0, 669
Telemetry [baseline] (8.754 ms) : 0, 8754
Telemetry [candidate] (8.944 ms) : 0, 8944
IAST [baseline] (21.267 ms) : 0, 21267
IAST [candidate] (21.784 ms) : 0, 21784
section profiling
BytebuddyAgent [baseline] (704.945 ms) : 0, 704945
BytebuddyAgent [candidate] (708.692 ms) : 0, 708692
GlobalTracer [baseline] (374.469 ms) : 0, 374469
GlobalTracer [candidate] (377.534 ms) : 0, 377534
AppSec [baseline] (54.662 ms) : 0, 54662
AppSec [candidate] (54.847 ms) : 0, 54847
Remote Config [baseline] (668.265 µs) : 0, 668
Remote Config [candidate] (676.324 µs) : 0, 676
Telemetry [baseline] (8.118 ms) : 0, 8118
Telemetry [candidate] (8.136 ms) : 0, 8136
ProfilingAgent [baseline] (96.077 ms) : 0, 96077
ProfilingAgent [candidate] (96.871 ms) : 0, 96871
Profiling [baseline] (96.1 ms) : 0, 96100
Profiling [candidate] (96.895 ms) : 0, 96895
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1059221
Total [baseline] (8.636 s) : 0, 8636217
Agent [candidate] (1.064 s) : 0, 1064131
Total [candidate] (8.668 s) : 0, 8667572
section iast
Agent [baseline] (1.19 s) : 0, 1189842
Total [baseline] (9.227 s) : 0, 9227332
Agent [candidate] (1.185 s) : 0, 1185299
Total [candidate] (9.221 s) : 0, 9220515
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.184 s) : 0, 1183994
Total [baseline] (9.236 s) : 0, 9236072
Agent [candidate] (1.187 s) : 0, 1187354
Total [candidate] (9.183 s) : 0, 9182784
section iast_TELEMETRY_OFF
Agent [baseline] (1.176 s) : 0, 1176157
Total [baseline] (9.209 s) : 0, 9209247
Agent [candidate] (1.187 s) : 0, 1187354
Total [candidate] (9.279 s) : 0, 9279265
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (715.482 ms) : 0, 715482
BytebuddyAgent [candidate] (718.076 ms) : 0, 718076
GlobalTracer [baseline] (256.162 ms) : 0, 256162
GlobalTracer [candidate] (258.309 ms) : 0, 258309
AppSec [baseline] (57.482 ms) : 0, 57482
AppSec [candidate] (57.142 ms) : 0, 57142
Remote Config [baseline] (720.246 µs) : 0, 720
Remote Config [candidate] (731.012 µs) : 0, 731
Telemetry [baseline] (14.46 ms) : 0, 14460
Telemetry [candidate] (14.656 ms) : 0, 14656
section iast
BytebuddyAgent [baseline] (837.249 ms) : 0, 837249
BytebuddyAgent [candidate] (832.588 ms) : 0, 832588
GlobalTracer [baseline] (248.363 ms) : 0, 248363
GlobalTracer [candidate] (248.652 ms) : 0, 248652
AppSec [baseline] (58.37 ms) : 0, 58370
AppSec [candidate] (58.324 ms) : 0, 58324
Remote Config [baseline] (661.299 µs) : 0, 661
Remote Config [candidate] (653.78 µs) : 0, 654
Telemetry [baseline] (8.794 ms) : 0, 8794
Telemetry [candidate] (8.866 ms) : 0, 8866
IAST [baseline] (21.314 ms) : 0, 21314
IAST [candidate] (21.181 ms) : 0, 21181
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (832.521 ms) : 0, 832521
BytebuddyAgent [candidate] (835.415 ms) : 0, 835415
GlobalTracer [baseline] (247.132 ms) : 0, 247132
GlobalTracer [candidate] (248.084 ms) : 0, 248084
AppSec [baseline] (58.339 ms) : 0, 58339
AppSec [candidate] (58.327 ms) : 0, 58327
Remote Config [baseline] (667.105 µs) : 0, 667
Remote Config [candidate] (646.681 µs) : 0, 647
Telemetry [baseline] (8.866 ms) : 0, 8866
Telemetry [candidate] (8.688 ms) : 0, 8688
IAST [baseline] (21.409 ms) : 0, 21409
IAST [candidate] (21.015 ms) : 0, 21015
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (826.603 ms) : 0, 826603
BytebuddyAgent [candidate] (834.83 ms) : 0, 834830
GlobalTracer [baseline] (246.702 ms) : 0, 246702
GlobalTracer [candidate] (248.783 ms) : 0, 248783
AppSec [baseline] (57.862 ms) : 0, 57862
AppSec [candidate] (58.374 ms) : 0, 58374
Remote Config [baseline] (653.348 µs) : 0, 653
Remote Config [candidate] (657.488 µs) : 0, 657
Telemetry [baseline] (8.569 ms) : 0, 8569
Telemetry [candidate] (8.668 ms) : 0, 8668
IAST [baseline] (20.769 ms) : 0, 20769
IAST [candidate] (20.776 ms) : 0, 20776
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section baseline
no_agent (1.341 ms) : 1321, 1361
. : milestone, 1341,
appsec (1.727 ms) : 1704, 1750
. : milestone, 1727,
appsec_no_iast (1.769 ms) : 1746, 1792
. : milestone, 1769,
iast (1.472 ms) : 1449, 1495
. : milestone, 1472,
profiling (1.505 ms) : 1481, 1530
. : milestone, 1505,
tracing (1.473 ms) : 1448, 1498
. : milestone, 1473,
section candidate
no_agent (1.344 ms) : 1324, 1365
. : milestone, 1344,
appsec (1.742 ms) : 1719, 1766
. : milestone, 1742,
appsec_no_iast (1.769 ms) : 1745, 1793
. : milestone, 1769,
iast (1.478 ms) : 1455, 1501
. : milestone, 1478,
profiling (1.485 ms) : 1461, 1508
. : milestone, 1485,
tracing (1.464 ms) : 1439, 1489
. : milestone, 1464,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section baseline
no_agent (375.015 µs) : 354, 396
. : milestone, 375,
iast (492.121 µs) : 470, 514
. : milestone, 492,
iast_FULL (646.172 µs) : 624, 668
. : milestone, 646,
iast_GLOBAL (518.436 µs) : 496, 541
. : milestone, 518,
iast_HARDCODED_SECRET_DISABLED (486.381 µs) : 465, 508
. : milestone, 486,
iast_INACTIVE (445.304 µs) : 424, 466
. : milestone, 445,
iast_TELEMETRY_OFF (471.681 µs) : 451, 493
. : milestone, 472,
tracing (448.772 µs) : 427, 470
. : milestone, 449,
section candidate
no_agent (369.508 µs) : 350, 389
. : milestone, 370,
iast (493.757 µs) : 472, 516
. : milestone, 494,
iast_FULL (643.213 µs) : 622, 665
. : milestone, 643,
iast_GLOBAL (527.492 µs) : 505, 550
. : milestone, 527,
iast_HARDCODED_SECRET_DISABLED (489.295 µs) : 468, 511
. : milestone, 489,
iast_INACTIVE (446.664 µs) : 426, 468
. : milestone, 447,
iast_TELEMETRY_OFF (483.942 µs) : 462, 506
. : milestone, 484,
tracing (447.974 µs) : 427, 469
. : milestone, 448,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section baseline
no_agent (15.43 s) : 15430000, 15430000
. : milestone, 15430000,
appsec (15.062 s) : 15062000, 15062000
. : milestone, 15062000,
iast (18.963 s) : 18963000, 18963000
. : milestone, 18963000,
iast_GLOBAL (18.256 s) : 18256000, 18256000
. : milestone, 18256000,
profiling (15.213 s) : 15213000, 15213000
. : milestone, 15213000,
tracing (14.812 s) : 14812000, 14812000
. : milestone, 14812000,
section candidate
no_agent (15.282 s) : 15282000, 15282000
. : milestone, 15282000,
appsec (14.929 s) : 14929000, 14929000
. : milestone, 14929000,
iast (18.989 s) : 18989000, 18989000
. : milestone, 18989000,
iast_GLOBAL (17.631 s) : 17631000, 17631000
. : milestone, 17631000,
profiling (15.097 s) : 15097000, 15097000
. : milestone, 15097000,
tracing (14.956 s) : 14956000, 14956000
. : milestone, 14956000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~75156f6274, baseline=1.45.0-SNAPSHOT~0b031046b3
dateFormat X
axisFormat %s
section baseline
no_agent (1.471 ms) : 1459, 1482
. : milestone, 1471,
appsec (2.347 ms) : 2305, 2390
. : milestone, 2347,
iast (2.112 ms) : 2058, 2166
. : milestone, 2112,
iast_GLOBAL (2.15 ms) : 2096, 2204
. : milestone, 2150,
profiling (1.958 ms) : 1915, 2001
. : milestone, 1958,
tracing (1.951 ms) : 1910, 1993
. : milestone, 1951,
section candidate
no_agent (1.473 ms) : 1462, 1485
. : milestone, 1473,
appsec (2.362 ms) : 2319, 2405
. : milestone, 2362,
iast (2.103 ms) : 2049, 2158
. : milestone, 2103,
iast_GLOBAL (2.154 ms) : 2100, 2209
. : milestone, 2154,
profiling (1.955 ms) : 1912, 1998
. : milestone, 1955,
tracing (1.949 ms) : 1906, 1991
. : milestone, 1949,
|
shatzi
reviewed
Jan 7, 2025
shatzi
left a comment
Contributor
There was a problem hiding this comment.
I think there are few things that are missing for me in this:
- the default rate limit seem to be off - 100/s for method doesn't seem high to me. specifically for handler methods that have <1% error rate..
- we don't report to the UI that we removed the instrumentation for the probe and let the user understand why the exception replay is not capturing specific frames.
- we don't report telemetry to keep track that this feature works.
- On one side, we are keeping instrumenting the hot frames if there multiple exception with the same frames. although we detect it was hot. I think we need to keep a list of hot frames - maybe even report that on a flare.
| if (exceptionProbeManager.shouldRemoveProbe(lastCaptureTime)) { | ||
| // hot loop without exception captured => uninstrument this probe | ||
| LOGGER.debug("Removing probe {}", id); | ||
| exceptionProbeManager.removeProbe(this); |
Contributor
There was a problem hiding this comment.
do we want to report this?
- maybe this should be trigger telemetry log event so we know how many times this happen to us.
- we we do capture the exception. should we report on that user frame as "HOT STUFF" so the UI would be able to display a message for the user why this frame was not captured?
| private final transient ExceptionProbeManager exceptionProbeManager; | ||
| private final transient int chainedExceptionIdx; | ||
| private final transient CircuitBreaker circuitBreaker = | ||
| new CircuitBreaker(1000, Duration.ofSeconds(HOT_LOOP_TIME_WINDOW_IN_SECONDS)); |
Contributor
There was a problem hiding this comment.
Do we want to make the 1000 configurable? currently any method that run more then 100 times a second will be filtered out.
How are we going to refine this value? what is the plan?
| @Override | ||
| public void evaluate( | ||
| CapturedContext context, CapturedContext.Status status, MethodLocation methodLocation) { | ||
| if (!circuitBreaker.trip()) { |
Contributor
There was a problem hiding this comment.
should this run after the check methodLocation != MethodLocation.EXIT ? or maybe it is not a thing... but if it is - then this trip() is get called twice?
jpbempel
commented
Jan 23, 2025
MemberAuthor
Withdrawn because superseded by #8230 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Add circuit breaker when evaluating exception probe to detect when an exception probe is executed in a hot path only waiting for exception to happen.
when circuit breaker is open after 1000 calls in 10 seconds window, we check if there is no exception captured during this period too and remove the probe from the configuration and un-instrument it. This will prevent too much overhead for hot path
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: DEBUG-3308