Add PR performance gate by bric3 · Pull Request #8970 · DataDog/dd-trace-java

bric3 · 2025-06-12T09:42:59Z

What Does This Do

Adds a PR Gate

Motivation

Additional Notes

Depends on

Fix benchmark failures and improve results quality #8962

Gate skippable via performance/ignore-performance-regression label, bp-runner PR still in progress.

=> https://datadoghq.atlassian.net/wiki/x/8YFzMwE

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-12T11:07:47Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/pr-gate
git_commit_date	1750861668	1750862595
git_commit_sha	`cb3d992`	`333b53a`
release_version	1.51.0-SNAPSHOT~cb3d9928f7	1.51.0-SNAPSHOT~333b53aa04

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750864479	1750864479
ci_job_id	998196561	998196561
ci_pipeline_id	68742349	68742349
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-a1vxyzsw-project-304-concurrent-1-4561qd6l 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-a1vxyzsw-project-304-concurrent-1-4561qd6l 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 8 unstable metrics.

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (996.401 ms) : 0, 996401
Total [baseline] (10.674 s) : 0, 10674365
Agent [candidate] (1.002 s) : 0, 1001530
Total [candidate] (10.775 s) : 0, 10775086
section appsec
Agent [baseline] (1.182 s) : 0, 1182397
Total [baseline] (10.903 s) : 0, 10903298
Agent [candidate] (1.181 s) : 0, 1180962
Total [candidate] (10.833 s) : 0, 10833404
section iast
Agent [baseline] (1.147 s) : 0, 1146960
Total [baseline] (10.912 s) : 0, 10911770
Agent [candidate] (1.138 s) : 0, 1137821
Total [candidate] (10.874 s) : 0, 10874421
section profiling
Agent [baseline] (1.252 s) : 0, 1252388
Total [baseline] (10.289 s) : 0, 10289185
Agent [candidate] (1.259 s) : 0, 1258987
Total [candidate] (11.099 s) : 0, 11099266

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	996.401 ms	-
Agent	appsec	1.182 s	185.996 ms (18.7%)
Agent	iast	1.147 s	150.56 ms (15.1%)
Agent	profiling	1.252 s	255.987 ms (25.7%)
Total	tracing	10.674 s	-
Total	appsec	10.903 s	228.933 ms (2.1%)
Total	iast	10.912 s	237.406 ms (2.2%)
Total	profiling	10.289 s	-385.179 ms (-3.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.002 s	-
Agent	appsec	1.181 s	179.432 ms (17.9%)
Agent	iast	1.138 s	136.292 ms (13.6%)
Agent	profiling	1.259 s	257.457 ms (25.7%)
Total	tracing	10.775 s	-
Total	appsec	10.833 s	58.318 ms (0.5%)
Total	iast	10.874 s	99.335 ms (0.9%)
Total	profiling	11.099 s	324.181 ms (3.0%)

gantt
title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (687.178 ms) : 0, 687178
BytebuddyAgent [candidate] (691.414 ms) : 0, 691414
GlobalTracer [baseline] (242.931 ms) : 0, 242931
GlobalTracer [candidate] (243.571 ms) : 0, 243571
AppSec [baseline] (30.407 ms) : 0, 30407
AppSec [candidate] (30.492 ms) : 0, 30492
Debugger [baseline] (6.054 ms) : 0, 6054
Debugger [candidate] (6.104 ms) : 0, 6104
Remote Config [baseline] (683.044 µs) : 0, 683
Remote Config [candidate] (679.191 µs) : 0, 679
Telemetry [baseline] (8.292 ms) : 0, 8292
Telemetry [candidate] (8.265 ms) : 0, 8265
section appsec
BytebuddyAgent [baseline] (714.862 ms) : 0, 714862
BytebuddyAgent [candidate] (713.977 ms) : 0, 713977
GlobalTracer [baseline] (237.834 ms) : 0, 237834
GlobalTracer [candidate] (237.747 ms) : 0, 237747
IAST [baseline] (22.208 ms) : 0, 22208
IAST [candidate] (22.137 ms) : 0, 22137
AppSec [baseline] (171.876 ms) : 0, 171876
AppSec [candidate] (171.536 ms) : 0, 171536
Debugger [baseline] (5.878 ms) : 0, 5878
Debugger [candidate] (5.845 ms) : 0, 5845
Remote Config [baseline] (619.202 µs) : 0, 619
Remote Config [candidate] (608.466 µs) : 0, 608
Telemetry [baseline] (8.208 ms) : 0, 8208
Telemetry [candidate] (8.158 ms) : 0, 8158
section iast
BytebuddyAgent [baseline] (818.644 ms) : 0, 818644
BytebuddyAgent [candidate] (812.037 ms) : 0, 812037
GlobalTracer [baseline] (236.28 ms) : 0, 236280
GlobalTracer [candidate] (234.497 ms) : 0, 234497
IAST [baseline] (28.193 ms) : 0, 28193
IAST [candidate] (28.847 ms) : 0, 28847
AppSec [baseline] (28.176 ms) : 0, 28176
AppSec [candidate] (26.975 ms) : 0, 26975
Debugger [baseline] (5.92 ms) : 0, 5920
Debugger [candidate] (5.904 ms) : 0, 5904
Remote Config [baseline] (589.345 µs) : 0, 589
Remote Config [candidate] (598.88 µs) : 0, 599
Telemetry [baseline] (8.065 ms) : 0, 8065
Telemetry [candidate] (7.975 ms) : 0, 7975
section profiling
BytebuddyAgent [baseline] (682.964 ms) : 0, 682964
BytebuddyAgent [candidate] (686.582 ms) : 0, 686582
GlobalTracer [baseline] (363.127 ms) : 0, 363127
GlobalTracer [candidate] (365.662 ms) : 0, 365662
AppSec [baseline] (32.655 ms) : 0, 32655
AppSec [candidate] (31.884 ms) : 0, 31884
Debugger [baseline] (11.105 ms) : 0, 11105
Debugger [candidate] (12.988 ms) : 0, 12988
Remote Config [baseline] (657.311 µs) : 0, 657
Remote Config [candidate] (667.301 µs) : 0, 667
Telemetry [baseline] (8.85 ms) : 0, 8850
Telemetry [candidate] (8.071 ms) : 0, 8071
ProfilingAgent [baseline] (104.106 ms) : 0, 104106
ProfilingAgent [candidate] (103.737 ms) : 0, 103737
Profiling [baseline] (104.13 ms) : 0, 104130
Profiling [candidate] (103.762 ms) : 0, 103762

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.005 s) : 0, 1004683
Total [baseline] (8.552 s) : 0, 8552071
Agent [candidate] (1.001 s) : 0, 1001094
Total [candidate] (8.63 s) : 0, 8630363
section iast
Agent [baseline] (1.133 s) : 0, 1132885
Total [baseline] (9.258 s) : 0, 9257600
Agent [candidate] (1.137 s) : 0, 1137057
Total [candidate] (9.313 s) : 0, 9313008

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.005 s	-
Agent	iast	1.133 s	128.203 ms (12.8%)
Total	tracing	8.552 s	-
Total	iast	9.258 s	705.529 ms (8.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.001 s	-
Agent	iast	1.137 s	135.963 ms (13.6%)
Total	tracing	8.63 s	-
Total	iast	9.313 s	682.645 ms (7.9%)

gantt
title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (693.765 ms) : 0, 693765
BytebuddyAgent [candidate] (690.07 ms) : 0, 690070
GlobalTracer [baseline] (244.134 ms) : 0, 244134
GlobalTracer [candidate] (244.227 ms) : 0, 244227
AppSec [baseline] (30.524 ms) : 0, 30524
AppSec [candidate] (30.717 ms) : 0, 30717
Debugger [baseline] (6.115 ms) : 0, 6115
Debugger [candidate] (6.152 ms) : 0, 6152
Remote Config [baseline] (680.316 µs) : 0, 680
Remote Config [candidate] (679.268 µs) : 0, 679
Telemetry [baseline] (8.309 ms) : 0, 8309
Telemetry [candidate] (8.386 ms) : 0, 8386
section iast
BytebuddyAgent [baseline] (809.097 ms) : 0, 809097
BytebuddyAgent [candidate] (811.752 ms) : 0, 811752
GlobalTracer [baseline] (233.402 ms) : 0, 233402
GlobalTracer [candidate] (233.732 ms) : 0, 233732
AppSec [baseline] (25.969 ms) : 0, 25969
AppSec [candidate] (29.686 ms) : 0, 29686
Debugger [baseline] (5.841 ms) : 0, 5841
Debugger [candidate] (5.855 ms) : 0, 5855
Remote Config [baseline] (593.248 µs) : 0, 593
Remote Config [candidate] (582.633 µs) : 0, 583
Telemetry [baseline] (7.925 ms) : 0, 7925
Telemetry [candidate] (8.076 ms) : 0, 8076
IAST [baseline] (29.281 ms) : 0, 29281
IAST [candidate] (26.43 ms) : 0, 26430

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/pr-gate
git_commit_date	1750861668	1750862595
git_commit_sha	`cb3d992`	`333b53a`
release_version	1.51.0-SNAPSHOT~cb3d9928f7	1.51.0-SNAPSHOT~333b53aa04

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750864174	1750864174
ci_job_id	998196562	998196562
ci_pipeline_id	68742349	68742349
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-a1vxyzsw-project-304-concurrent-2-b6j4i1ns 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-a1vxyzsw-project-304-concurrent-2-b6j4i1ns 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 11 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:tracing:high_load	worse [+1.334ms; +2.108ms] or [+3.061%; +4.836%]	unstable [-12.089op/s; +1.220op/s] or [-11.118%; +1.122%]	45.301ms	103.300op/s	43.581ms	108.734op/s

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section baseline
no_agent (37.658 ms) : 37352, 37964
. : milestone, 37658,
appsec (49.345 ms) : 48914, 49775
. : milestone, 49345,
code_origins (45.157 ms) : 44775, 45539
. : milestone, 45157,
iast (44.792 ms) : 44396, 45188
. : milestone, 44792,
profiling (48.683 ms) : 48238, 49128
. : milestone, 48683,
tracing (43.581 ms) : 43228, 43934
. : milestone, 43581,
section candidate
no_agent (37.394 ms) : 37090, 37698
. : milestone, 37394,
appsec (48.376 ms) : 47956, 48796
. : milestone, 48376,
code_origins (45.332 ms) : 44952, 45712
. : milestone, 45332,
iast (43.879 ms) : 43504, 44253
. : milestone, 43879,
profiling (47.966 ms) : 47528, 48404
. : milestone, 47966,
tracing (45.301 ms) : 44935, 45668
. : milestone, 45301,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.658 ms [37.352 ms, 37.964 ms]	-
appsec	49.345 ms [48.914 ms, 49.775 ms]	11.687 ms (31.0%)
code_origins	45.157 ms [44.775 ms, 45.539 ms]	7.499 ms (19.9%)
iast	44.792 ms [44.396 ms, 45.188 ms]	7.134 ms (18.9%)
profiling	48.683 ms [48.238 ms, 49.128 ms]	11.025 ms (29.3%)
tracing	43.581 ms [43.228 ms, 43.934 ms]	5.923 ms (15.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.394 ms [37.09 ms, 37.698 ms]	-
appsec	48.376 ms [47.956 ms, 48.796 ms]	10.982 ms (29.4%)
code_origins	45.332 ms [44.952 ms, 45.712 ms]	7.938 ms (21.2%)
iast	43.879 ms [43.504 ms, 44.253 ms]	6.485 ms (17.3%)
profiling	47.966 ms [47.528 ms, 48.404 ms]	10.572 ms (28.3%)
tracing	45.301 ms [44.935 ms, 45.668 ms]	7.907 ms (21.1%)

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section baseline
no_agent (4.409 ms) : 4359, 4460
. : milestone, 4409,
iast (8.949 ms) : 8806, 9092
. : milestone, 8949,
iast_FULL (14.111 ms) : 13830, 14393
. : milestone, 14111,
iast_GLOBAL (9.68 ms) : 9501, 9859
. : milestone, 9680,
profiling (8.593 ms) : 8461, 8726
. : milestone, 8593,
tracing (7.862 ms) : 7751, 7974
. : milestone, 7862,
section candidate
no_agent (4.392 ms) : 4343, 4442
. : milestone, 4392,
iast (9.264 ms) : 9111, 9418
. : milestone, 9264,
iast_FULL (13.763 ms) : 13489, 14037
. : milestone, 13763,
iast_GLOBAL (9.856 ms) : 9679, 10034
. : milestone, 9856,
profiling (8.87 ms) : 8732, 9008
. : milestone, 8870,
tracing (7.63 ms) : 7521, 7739
. : milestone, 7630,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.409 ms [4.359 ms, 4.46 ms]	-
iast	8.949 ms [8.806 ms, 9.092 ms]	4.54 ms (103.0%)
iast_FULL	14.111 ms [13.83 ms, 14.393 ms]	9.702 ms (220.0%)
iast_GLOBAL	9.68 ms [9.501 ms, 9.859 ms]	5.271 ms (119.5%)
profiling	8.593 ms [8.461 ms, 8.726 ms]	4.184 ms (94.9%)
tracing	7.862 ms [7.751 ms, 7.974 ms]	3.453 ms (78.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.392 ms [4.343 ms, 4.442 ms]	-
iast	9.264 ms [9.111 ms, 9.418 ms]	4.872 ms (110.9%)
iast_FULL	13.763 ms [13.489 ms, 14.037 ms]	9.371 ms (213.3%)
iast_GLOBAL	9.856 ms [9.679 ms, 10.034 ms]	5.464 ms (124.4%)
profiling	8.87 ms [8.732 ms, 9.008 ms]	4.478 ms (101.9%)
tracing	7.63 ms [7.521 ms, 7.739 ms]	3.238 ms (73.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/pr-gate
git_commit_date	1750861668	1750862595
git_commit_sha	`cb3d992`	`333b53a`
release_version	1.51.0-SNAPSHOT~cb3d9928f7	1.51.0-SNAPSHOT~333b53aa04

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1750864690	1750864690
ci_job_id	998196563	998196563
ci_pipeline_id	68742349	68742349
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-a1vxyzsw-project-304-concurrent-3-sbdoj3ed 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-a1vxyzsw-project-304-concurrent-3-sbdoj3ed 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section baseline
no_agent (1.473 ms) : 1461, 1484
. : milestone, 1473,
appsec (2.406 ms) : 2357, 2455
. : milestone, 2406,
iast (2.184 ms) : 2122, 2245
. : milestone, 2184,
iast_GLOBAL (2.226 ms) : 2164, 2288
. : milestone, 2226,
profiling (2.049 ms) : 1999, 2100
. : milestone, 2049,
tracing (2.006 ms) : 1958, 2053
. : milestone, 2006,
section candidate
no_agent (1.473 ms) : 1461, 1484
. : milestone, 1473,
appsec (2.405 ms) : 2356, 2454
. : milestone, 2405,
iast (2.193 ms) : 2132, 2254
. : milestone, 2193,
iast_GLOBAL (2.226 ms) : 2165, 2288
. : milestone, 2226,
profiling (2.029 ms) : 1980, 2079
. : milestone, 2029,
tracing (2.009 ms) : 1961, 2056
. : milestone, 2009,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.461 ms, 1.484 ms]	-
appsec	2.406 ms [2.357 ms, 2.455 ms]	933.344 µs (63.4%)
iast	2.184 ms [2.122 ms, 2.245 ms]	711.172 µs (48.3%)
iast_GLOBAL	2.226 ms [2.164 ms, 2.288 ms]	753.531 µs (51.2%)
profiling	2.049 ms [1.999 ms, 2.1 ms]	576.671 µs (39.2%)
tracing	2.006 ms [1.958 ms, 2.053 ms]	532.806 µs (36.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.461 ms, 1.484 ms]	-
appsec	2.405 ms [2.356 ms, 2.454 ms]	932.154 µs (63.3%)
iast	2.193 ms [2.132 ms, 2.254 ms]	720.506 µs (48.9%)
iast_GLOBAL	2.226 ms [2.165 ms, 2.288 ms]	753.858 µs (51.2%)
profiling	2.029 ms [1.98 ms, 2.079 ms]	556.747 µs (37.8%)
tracing	2.009 ms [1.961 ms, 2.056 ms]	535.934 µs (36.4%)

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~333b53aa04, baseline=1.51.0-SNAPSHOT~cb3d9928f7
dateFormat X
axisFormat %s
section baseline
no_agent (15.539 s) : 15539000, 15539000
. : milestone, 15539000,
appsec (15.133 s) : 15133000, 15133000
. : milestone, 15133000,
iast (18.353 s) : 18353000, 18353000
. : milestone, 18353000,
iast_GLOBAL (18.079 s) : 18079000, 18079000
. : milestone, 18079000,
profiling (15.017 s) : 15017000, 15017000
. : milestone, 15017000,
tracing (14.787 s) : 14787000, 14787000
. : milestone, 14787000,
section candidate
no_agent (15.08 s) : 15080000, 15080000
. : milestone, 15080000,
appsec (14.711 s) : 14711000, 14711000
. : milestone, 14711000,
iast (18.557 s) : 18557000, 18557000
. : milestone, 18557000,
iast_GLOBAL (17.691 s) : 17691000, 17691000
. : milestone, 17691000,
profiling (15.212 s) : 15212000, 15212000
. : milestone, 15212000,
tracing (14.845 s) : 14845000, 14845000
. : milestone, 14845000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.539 s [15.539 s, 15.539 s]	-
appsec	15.133 s [15.133 s, 15.133 s]	-406.0 ms (-2.6%)
iast	18.353 s [18.353 s, 18.353 s]	2.814 s (18.1%)
iast_GLOBAL	18.079 s [18.079 s, 18.079 s]	2.54 s (16.3%)
profiling	15.017 s [15.017 s, 15.017 s]	-522.0 ms (-3.4%)
tracing	14.787 s [14.787 s, 14.787 s]	-752.0 ms (-4.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.08 s [15.08 s, 15.08 s]	-
appsec	14.711 s [14.711 s, 14.711 s]	-369.0 ms (-2.4%)
iast	18.557 s [18.557 s, 18.557 s]	3.477 s (23.1%)
iast_GLOBAL	17.691 s [17.691 s, 17.691 s]	2.611 s (17.3%)
profiling	15.212 s [15.212 s, 15.212 s]	132.0 ms (0.9%)
tracing	14.845 s [14.845 s, 14.845 s]	-235.0 ms (-1.6%)

…ion.yml path

…on-regression.yml path

bric3 · 2025-06-18T15:07:06Z

todo: From @ddyurchenko
Could you please rebase your PR on the latest master and remove allow_failure: true from gating job?
In worst case scenario if we block something that should not be blocked, there is a label mechanism to bypass the gate
Ignore mechanism in bp-runner

Thanks for flagging it! We need to remove allow_failure: true, the bypass mechanism inside bp-runner is specified in https://datadoghq.atlassian.net/wiki/x/LgI1LgE#How-to-bypass%3F.

…efined)

ddyurchenko

Looks good! Both cases where gates should block and where they should let pass work. ✔️

PerfectSlayer

Can you link doc about it?

Gate skippable via performance/ignore-performance-regression label

Is that a GitHub issue label? If so, is it configurable as it does not fit our usual pattern?

bric3 · 2025-06-19T10:25:51Z

@PerfectSlayer The bypass link is mentioned in @ddyurchenko's comment, I added it in the PR description.

Is that a GitHub issue label?

Yes.

If so, is it configurable as it does not fit our usual pattern?

I already brought this to @ddyurchenko, this is something to be addressed in bp-runner PR (which is not merged yet). (Hence the higher threshold at this time).

bric3 · 2025-06-25T13:34:28Z

There are still some concerns about the variability of the benchmarks.

E.g. this PR that sees 9% worse while touching unrelated files.
worse
[+3.367ms; +4.340ms] or [+7.002%; +9.027%]
Cancel GitLab running pipeline on new PR push #9023
Or this PR that see 8% improvement
better
[-379.158µs; -273.744µs] or [-8.414%; -6.075%]
Convert some build file to kotlin dsl #8743

20% was chosen due to variability of the results on unrelated changes. After discussion in lp-sync, 20% won't get us to move forward and entice to fix the problems. So instead choose a closer threshold to 10%.

bric3 added tag: performance Performance related changes tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jun 12, 2025

bric3 requested a review from a teamJune 12, 2025 09:46

ddyurchenko reviewed Jun 12, 2025
View reviewed changes

Comment thread.gitlab/benchmarks.yml Outdated

ddyurchenko reviewed Jun 12, 2025
View reviewed changes

Comment thread.gitlab/benchmarks.yml Outdated

bric3 force-pushed the bdu/pr-gate branch from 89c9ea7 to 301e9f8Compare June 12, 2025 11:38

bric3 marked this pull request as ready for review June 13, 2025 16:17

bric3 requested a review from a team as a code owner June 13, 2025 16:17

bric3 requested review from colin-higgins and removed request for a teamJune 13, 2025 16:17

bric3 force-pushed the bdu/pr-gate branch from 1deb0cd to f76655fCompare June 13, 2025 19:54

PerfectSlayer changed the title ~~feat(ci): PR performance gate~~Add PR performance gateJun 14, 2025

PerfectSlayer added the type: feature Enhancements and improvements label Jun 14, 2025

bric3 force-pushed the bdu/pr-gate branch 3 times, most recently from f806916 to 4aeb1c8Compare June 18, 2025 14:50

bric3 added 7 commits June 18, 2025 17:05

feat(ci): PR performance gate
51cdcd5

chore(ci): Review comments
1e75a62

feat(ci): Explicitly rely on job artifacts, drop dacapo
109573f

fix(ci): revert removal of ARTIFACTSDIR for bp-runner.fail-on-regress…
3f56fdf
…ion.yml path

chore(ci): replace ARTIFACTSDIR by CI_PROJECT_DIR for bp-runner.fail-…
52b5b58
…on-regression.yml path

chore(ci): Move threshold to 10%
4018257

chore(ci): Allow failure on PR gate to exercise it
e6af269

bric3 commented Jun 18, 2025
View reviewed changes

chore(ci): Disable failure but increase regression threshold (to be r…
5bf274c
…efined)

bric3 force-pushed the bdu/pr-gate branch from 4aeb1c8 to 5bf274cCompare June 18, 2025 15:09

ddyurchenko approved these changes Jun 18, 2025
View reviewed changes

PerfectSlayer reviewed Jun 19, 2025
View reviewed changes

bric3 added 2 commits June 24, 2025 12:19

Merge remote-tracking branch 'origin/master' into bdu/pr-gate
59268cd

Merge remote-tracking branch 'origin/master' into bdu/pr-gate
460c5e4

bric3 added 2 commits June 25, 2025 16:29

Merge remote-tracking branch 'origin/master' into bdu/pr-gate
417db44

chore(ci): Choose 10% as threshold.
333b53a
20% was chosen due to variability of the results on unrelated changes. After discussion in lp-sync, 20% won't get us to move forward and entice to fix the problems. So instead choose a closer threshold to 10%.

bric3 enabled auto-merge (squash) June 25, 2025 14:43

bric3 merged commit d6d3d21 into masterJun 25, 2025

bric3 deleted the bdu/pr-gate branch June 25, 2025 15:36

github-actionsBot added this to the 1.51.0 milestone Jun 25, 2025

Navigation Menu

Uh oh!

Add PR performance gate - #8970

Add PR performance gate#8970
bric3 merged 12 commits into
masterfrom
bdu/pr-gate

bric3 commented Jun 12, 2025•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

pr-commenterBot commented Jun 12, 2025•
edited
Loading

Uh oh!

bric3Jun 18, 2025

Uh oh!

ddyurchenkoJun 18, 2025•
edited by bric3
Loading

Uh oh!

ddyurchenko left a comment

Uh oh!

PerfectSlayer left a comment

Uh oh!

bric3 commented Jun 19, 2025

Uh oh!

bric3 commented Jun 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

bric3 commented Jun 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

Uh oh!

Uh oh!

pr-commenterBot commented Jun 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

bric3Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

ddyurchenkoJun 18, 2025• edited by bric3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddyurchenko left a comment

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

bric3 commented Jun 19, 2025

Uh oh!

bric3 commented Jun 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bric3 commented Jun 12, 2025•
edited
Loading

pr-commenterBot commented Jun 12, 2025•
edited
Loading

ddyurchenkoJun 18, 2025•
edited by bric3
Loading