Skip to content

Replace Logging Piccolo benchmark with Locust - #8206

Open
Amaury Chamayou (achamayou) wants to merge 25 commits into
mainfrom
achamayou-logging-locust-benchmark
Open

Replace Logging Piccolo benchmark with Locust#8206
Amaury Chamayou (achamayou) wants to merge 25 commits into
mainfrom
achamayou-logging-locust-benchmark

Conversation

@achamayou

@achamayouAmaury Chamayou (achamayou) commented Aug 24, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • replace the certificate-authenticated Piccolo pi_ls benchmark with pi_logging_cert_blocking_locust
  • send user-certificate-authenticated POST /app/log/blocking/private requests and wait for commit before each Locust user sends its next request
  • reuse Replace Logging JWT perf test with Locust #8201's shared Locust orchestration and one logging workload implementation for certificate and JWT authentication
  • run 640 users at a 640 users/s spawn rate across 10 Locust processes, measure for 20 seconds at a 20ms signature interval, and write across 1,000 record IDs
  • publish the result under the distinct Logging Certificate Blocking Locust Bencher label

Depends on #8201.

The old Logging result measured 100,000 pipelined requests to the non-blocking /app/log/private endpoint with Piccolo and --max-writes-ahead 1000. This replacement keeps normal user-certificate authentication, but measures one outstanding blocking-on-commit request per Locust user. The results are not directly comparable, so this does not reuse the old Logging label.

Validation

  • configured CMake with BUILD_TESTS=ON and WORKER_THREADS=2, then inspected the generated certificate and JWT CTest commands
  • parsed both benchmark CLIs and discovered the shared workload with Locust in certificate and JWT modes
  • exercised certificate selection, TLS certificate/key propagation, request endpoint/body/headers, and preserved JWT bearer handling in a focused Python check
  • compiled the changed Python files with py_compile
  • ran Black, Ruff, gersemi, and the repository ASCII check
  • ran two client-count sweeps in the Benchmark A/B workflow:
UsersThroughputp50 latencyp99 latencyImplied mean latencyMemory
32015,079.2 tx/s20 ms26 ms21.2 ms90.7 MiB
48022,006.6 tx/s20 ms24 ms21.8 ms102.7 MiB
56024,984.2 tx/s20 ms27 ms22.4 ms110.9 MiB
64025,193.8 tx/s21 ms37 ms25.4 ms117.7 MiB
72024,312.7 tx/s23 ms42 ms29.6 ms125.3 MiB
80024,270.6 tx/s27 ms46 ms33.0 ms132.1 MiB
90018,929.5 tx/s44 ms72 ms47.5 ms140.2 MiB
1,00016,704.5 tx/s62 ms97 ms59.9 ms152.4 MiB

Throughput and latency by concurrent users

The curve has a broad 560-640-user plateau rather than a sharp peak: 560 users already reach 99.2% of the 640-user throughput. Each user keeps one blocking request outstanding with no think time, so below saturation throughput is approximately the client count divided by the roughly 20ms commit latency. Around 560 users, the service reaches its approximately 25,000 tx/s capacity. Additional users then add queueing rather than useful work: implied mean latency rises from 22.4ms at 560 users to 33.0ms at 800, then requests increasingly wait across multiple commit intervals at 900 and 1,000 users while throughput falls.

The benchmark keeps 640 users because it reliably saturates the service with some headroom against run-to-run variation. Two separate single-count 640-user runs reproduced 24,922.5 and 25,381.6 tx/s, both with 21ms p50 and 37ms p99.

@achamayou
Replace Logging JWT perf test with Locust
e528a06
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Merge branch 'main' into achamayou-logging-jwt-locust-benchmark
d510c07
@achamayou
Validate Locust spawn rate
b10b8a8
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Strengthen Locust benchmark validation tests
50eeb02
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Remove Locust infrastructure tests
3a1a81d
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Trim redundant benchmark logging
cd566c8
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Use one signing interval for JWT benchmark
e1b0e33
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Merge branch 'main' into achamayou-logging-jwt-locust-benchmark
8af0ff2
@achamayou
Replace Logging Piccolo benchmark with Locust
e0d7832
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Base automatically changed from achamayou-logging-jwt-locust-benchmark to mainAugust 25, 2026 14:57
@achamayou
Merge main into Logging Locust benchmark
1d0d10e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 893d95e8-6329-4be1-a1b4-7740b8a9d97e
@github-actions

github-actionsBot commented Aug 25, 2026•
edited
Loading

Copy link
Copy Markdown
Description

Comparing 5 available runs from this branch (#8206) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 5 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(8){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 68,585 tx/s ▬ -2%"]
axis b2["Basic Blocking Locust 100ms: 3,052 tx/s ▬ 0%"]
axis b3["Basic Blocking Locust 20ms: 15,193 tx/s ▬ 0%"]
axis b4["Basic Blocking Locust 2ms: 31,935 tx/s ▬ 0%"]
axis b5["Basic JS: 4,793 tx/s ▬ +1%"]
axis b7["Historical Queries: 214,335 tx/s ▬ +3%"]
axis b9["L…g C…e Blocking Locust: 25,120 tx/s ▲ 3%"]
axis b11["L…g JWT Blocking Locust: 14,836 tx/s ▬ +1%"]
curve stddev2_high["main EWMA + 2 std dev"]{106.93, 100.54, 100.56, 109.23, 105.73, 108.73, 105.78, 101.94}
curve stddev1_high["main EWMA + 1 std dev"]{103.46, 100.27, 100.28, 104.61, 102.87, 104.37, 102.89, 100.97}
curve stddev1_low["main EWMA - 1 std dev"]{96.54, 99.73, 99.72, 95.39, 97.13, 95.63, 97.11, 99.03}
curve stddev2_low["main EWMA - 2 std dev"]{93.07, 99.46, 99.44, 90.77, 94.27, 91.27, 94.22, 98.06}
curve branch_0["#8206 (4 runs earlier)"]{97.85, 100.26, 100.08, 89.96, 101.80, 101.38, 100.00, 98.43}
curve branch_1["#8206 (3 runs earlier)"]{101.97, 100.45, 100.02, 101.83, 100.43, 97.83, 99.86, 101.32}
curve branch_2["#8206 (2 runs earlier)"]{98.68, 99.76, 99.59, 94.86, 97.50, 93.35, 94.47, 100.11}
curve branch_3["#8206 (1 run earlier)"]{101.62, 100.50, 100.11, 102.50, 101.63, 101.41, 100.69, 101.08}
curve branch_4["#8206"]{97.63, 100.05, 100.18, 100.15, 100.80, 103.12, 103.35, 100.50}
graticule polygon
max 116
min 83
ticks 0
showLegend false
Loading

Latency (ms)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(8){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking Locust 100ms: 99 ms ▬ 0%"]
axis b1["Basic Blocking Locust 20ms: 19 ms ▬ 0%"]
axis b2["Basic Blocking Locust 2ms: 9 ms ▬ +1%"]
axis b3["Commit Latency 16ms: 4 ms ▬ -17%"]
axis b4["Commit Latency 1ms: 1.95 ms ▬ +1%"]
axis b5["Commit Latency 256ms: 215 ms ▬ 0%"]
axis b6["L…g Certificate Blocking Locust: 21 ms ▼ 5%"]
axis b7["Logging JWT Blocking Locust: 20 ms ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 109.12, 148.92, 105.54, 102.36, 104.45, 100.00}
curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 104.56, 124.46, 102.77, 101.18, 102.23, 100.00}
curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 95.44, 75.54, 97.23, 98.82, 97.77, 100.00}
curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 90.88, 51.08, 94.46, 97.64, 95.55, 100.00}
curve branch_0["#8206 (4 runs earlier)"]{100.00, 100.00, 112.03, 91.08, 98.83, 100.43, 100.00, 100.00}
curve branch_1["#8206 (3 runs earlier)"]{100.00, 100.00, 100.83, 88.73, 100.81, 99.75, 95.45, 100.00}
curve branch_2["#8206 (2 runs earlier)"]{100.00, 100.00, 100.83, 139.63, 102.36, 99.77, 100.00, 100.00}
curve branch_3["#8206 (1 run earlier)"]{100.00, 100.00, 100.83, 99.75, 101.92, 100.18, 100.00, 100.00}
curve branch_4["#8206"]{100.00, 100.00, 100.83, 83.07, 101.23, 100.46, 95.45, 100.00}
graticule polygon
max 184
min 16
ticks 0
showLegend false
Loading

Memory (bytes)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#E5484D!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 91.4 MiB ▲ 4%"]
axis b2["Basic Blocking Locust 100ms: 89.9 MiB ▬ -1%"]
axis b3["Basic Blocking Locust 20ms: 93.3 MiB ▲ 3%"]
axis b4["Basic Blocking Locust 2ms: 93.4 MiB ▬ 0%"]
axis b5["Basic JS: 71.2 MiB ▬ 0%"]
axis b8["L…g C…e Blocking Locust: 120 MiB ▬ 0%"]
axis b10["Logging JWT Blocking Locust: 93.6 MiB ▲ 2%"]
curve stddev2_high["main EWMA + 2 std dev"]{105.40, 101.97, 101.62, 101.97, 103.43, 101.37, 102.40}
curve stddev1_high["main EWMA + 1 std dev"]{102.70, 100.99, 100.81, 100.98, 101.71, 100.68, 101.20}
curve stddev1_low["main EWMA - 1 std dev"]{97.30, 99.01, 99.19, 99.02, 98.29, 99.32, 98.80}
curve stddev2_low["main EWMA - 2 std dev"]{94.60, 98.03, 98.38, 98.03, 96.57, 98.63, 97.60}
curve branch_0["#8206 (4 runs earlier)"]{103.45, 98.74, 99.01, 99.34, 102.78, 100.00, 99.72}
curve branch_1["#8206 (3 runs earlier)"]{98.22, 100.47, 100.03, 99.07, 98.41, 99.06, 98.31}
curve branch_2["#8206 (2 runs earlier)"]{103.53, 101.28, 99.63, 99.32, 99.27, 101.09, 99.94}
curve branch_3["#8206 (1 run earlier)"]{99.69, 99.39, 99.83, 100.85, 98.60, 99.74, 99.09}
curve branch_4["#8206"]{103.65, 99.22, 102.80, 99.58, 100.31, 100.47, 102.27}
graticule polygon
max 110
min 90
ticks 0
showLegend false
Loading

Rate (ops/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["CHAMP get: 37,975,153 ops/s ▬ 0%"]
axis b1["CHAMP put: 5,500,600 ops/s ▬ 0%"]
axis b2["KV deserialisation: 1,647,718 ops/s ▬ 0%"]
axis b3["KV serialisation: 1,386,770 ops/s ▬ -1%"]
axis b4["KV s…t deserialisation: 4,213 ops/s ▬ 0%"]
axis b5["KV snapshot serialisation: 4,374 ops/s ▬ -2%"]
curve stddev2_high["main EWMA + 2 std dev"]{102.36, 103.36, 105.39, 106.44, 104.10, 111.22}
curve stddev1_high["main EWMA + 1 std dev"]{101.18, 101.68, 102.69, 103.22, 102.05, 105.61}
curve stddev1_low["main EWMA - 1 std dev"]{98.82, 98.32, 97.31, 96.78, 97.95, 94.39}
curve stddev2_low["main EWMA - 2 std dev"]{97.64, 96.64, 94.61, 93.56, 95.90, 88.78}
curve branch_0["#8206 (4 runs earlier)"]{98.44, 99.88, 98.55, 100.92, 99.39, 97.68}
curve branch_1["#8206 (3 runs earlier)"]{100.25, 100.41, 100.53, 99.07, 101.09, 102.10}
curve branch_2["#8206 (2 runs earlier)"]{100.63, 99.17, 99.05, 96.91, 99.02, 92.72}
curve branch_3["#8206 (1 run earlier)"]{99.99, 100.73, 98.25, 99.09, 101.00, 102.27}
curve branch_4["#8206"]{99.99, 100.35, 100.19, 98.66, 100.40, 97.60}
graticule polygon
max 120
min 80
ticks 0
showLegend false
Loading

@achamayou
Amaury Chamayou (achamayou) marked this pull request as ready for review August 25, 2026 19:44
CopilotAI lite review requested due to automatic review settings August 25, 2026 19:44
CopilotAI reviewed Aug 25, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates CCF’s performance benchmarking suite by replacing the legacy Piccolo-based certificate-authenticated logging benchmark with a Locust-based benchmark, aligning it with the newer shared Locust orchestration and workload approach (as introduced in the dependent PR #8201).

Changes:

  • Added a new certificate-authenticated Locust benchmark entrypoint (tests/logging_locust.py) for blocking-on-commit logging writes.
  • Updated the JWT logging benchmark entrypoint to reuse the shared logging_locustfile.py workload and explicitly select JWT authentication via CLI.
  • Extended the shared logging Locust workload to support both certificate and JWT authentication modes, and updated CMake to register the new perf test target and Bencher label.

Custom instructions used:

  • .github/copilot-instructions.md

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

FileDescription
tests/logging_locust.pyNew Locust benchmark entrypoint for user-certificate-authenticated blocking logging writes.
tests/logging_jwt_locust.pySwitches JWT benchmark to the shared logging locustfile and passes explicit --authentication jwt.
tests/infra/logging_locustfile.pyGeneralizes the logging Locust workload to support both certificate and JWT authentication.
CMakeLists.txtReplaces the old Piccolo pi_ls perf test with a Locust-driven pi_logging_cert_blocking_locust perf test and new Bencher label.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@achamayou
Sweep Locust certificate client counts
b14e349
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Tolerate slow benchmark package feeds
43a5819
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Use 640 clients for logging benchmark
fe39053
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Sweep Locust saturation knee
9750bc7
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Restore 640-client benchmark
c25ddf1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Scan logging with three workers
141705b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Restore two-worker benchmark workflow
9c95e9a
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayouAmaury Chamayou (achamayou) mentioned this pull request Aug 27, 2026
Closed
@achamayouAmaury Chamayou (achamayou) mentioned this pull request Aug 27, 2026
Open
@achamayouAmaury Chamayou (achamayou) mentioned this pull request Aug 27, 2026
Open
eddyashton
Eddy Ashton (eddyashton) reviewed Aug 28, 2026
View reviewed changes
Comment threadtests/logging_cert_locust.py
eddyashton
Eddy Ashton (eddyashton) reviewed Aug 28, 2026
View reviewed changes
Comment threadtests/infra/logging_locustfile.py Outdated
@achamayou
Rename logging_locust.py to logging_cert_locust.py
16ad8cf
Mutual cert auth is one of several authentication options rather than the
default, so name the driver after the authentication it exercises, matching
logging_jwt_locust.py.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Drop pi_ prefix from locust-driven perf tests
4e1f508
These are add_e2e_test targets driven by Locust, not add_piccolo_test
targets, so the piccolo prefix is misleading. CI selects them with
-L perf -C perf and Bencher keys metrics on --perf-label, both unchanged.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Select logging authentication with a subcommand
b72ffb0
argparse now enforces that --cert and --key are supplied together, and only
under cert authentication, replacing the manual checks in BlockingWriter with
uniform argparse errors.
Everything after a subcommand is parsed by its subparser, so run_locust now
appends the workload arguments last.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Declare the logging benchmark JWT as an option
907d433
The token still only reaches Locust through the environment, but reading it
via env_var means argparse requires it, rather than BlockingWriter raising
once users start.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eddyashtonEddy Ashton (eddyashton)eddyashton approved these changes
Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

bench-ab

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@achamayou@eddyashton