Replace Basic JS Piccolo benchmark with Locust

[Amaury Chamayou (achamayou)]

Summary

• replace the Piccolo-driven pi_basic_js benchmark with basic_js_locust
• reuse the shared Basic Locust workload, selecting PUT /records/{key} for the JavaScript app while preserving the native app's blocking endpoint
• run 320 users at a 320 users/s spawn rate across 10 Locust processes, measure for 20 seconds, and retain the old benchmark's 100ms signing interval
• publish results under the distinct Basic JS Locust Bencher label because the sequential Locust workload is not directly comparable to Piccolo's pipelined workload

Inspired by #8206.

--endpoint takes a path

The locustfile takes the endpoint path verbatim, with {key} substituted per request, rather than a key into a table of known paths. The paths live at the calling point:

BLOCKING_ENDPOINT="/records/blocking/{key}"RECORDS_ENDPOINT="/records/{key}"

Any endpoint can therefore be benchmarked without editing the locustfile, which makes manual one-off runs much easier. Note that Locust never showed the previous choices list in --help anyway: it parses -f and short-circuits on --help before importing the locustfile, so init_command_line_parser has no listeners registered yet and none of this locustfile's arguments appear there.

The JavaScript workload is not blocking

The native workload uses PUT /records/blocking/{key}, which only responds once the transaction has committed. The JavaScript workload uses PUT /records/{key}, which responds without waiting for consensus commit, so the two throughput figures are not directly comparable and are published under separate Bencher labels.

This is a framework limitation rather than a choice. Responding on commit is a C++-only API:

• it is driven by ctx.rpc_ctx->set_consensus_committed_function(...) (include/ccf/rpc_context.h), which samples/apps/basic/basic.cpp uses to install /records/blocking/{key}
• src/js/registry.cpp never calls it
• EndpointProperties, which is what app.json maps to, has no corresponding field
• the TypeScript API in js/ccf-app exposes only getLastCommittedTxId

Giving the JavaScript app a blocking endpoint would mean adding respond-on-commit support to the JS runtime, which is out of scope here. The reason is recorded in the benchmark's docstring and at the branch point so it does not have to be rediscovered.

Validation

• configured CMake with BUILD_TESTS=ON and WORKER_THREADS=2 and inspected the generated command
• parsed the benchmark CLI and discovered the Writer workload with Locust
• ran the workload against a local server that records the paths it receives, confirming the default yields /records/blocking/0..3, --endpoint "/records/{key}" yields /records/0..3, and an arbitrary --endpoint "/app/anything/{key}/suffix" yields /app/anything/0..3/suffix, each aggregated under a single correctly named statistics entry
• confirmed the pi_ rename is inert: CI selects perf tests with -L perf -C perf, Bencher keys metrics on the unchanged --perf-label, and no workflow, script, doc or tests/ci-buckets.txt entry references the test name
• ran Black, Ruff, gersemi, copyright, ASCII, and Python compilation checks

[Copilot]

Pull request overview

This PR replaces the Piccolo-driven pi_basic_js performance benchmark with a Locust-based benchmark (pi_basic_js_locust), reusing the shared Basic Locust workload while selecting the appropriate endpoint variant for C++ vs JavaScript basic apps. This modernizes the benchmark harness to make concurrency tunable and results easier to interpret (with a distinct Bencher label due to non-comparability with Piccolo’s pipelining).

Changes:

• Add a new pi_basic_js_locust CTest target using tests/basicperf_locust.py and publish results under the Basic JS Locust label.
• Extend the shared Basic Locust workload to support endpoint selection (/records/blocking/{key} vs /records/{key}) and pick the variant based on whether a JS app bundle is used.
• Generalize Locust benchmark orchestration/help text to reflect non-blocking as well as blocking workloads.

Custom instructions used:

• None (no files from .github/copilot-instructions.md or .github/instructions/ were explicitly loaded during this review).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/infra/locust_benchmark.py	Generalizes shared Locust benchmark orchestration and standardizes consensus-update interval behavior/comments.
tests/infra/locust_benchmark_support.py	Broadens helper module docstring from “blocking-only” to generic benchmark workloads.
tests/infra/basicperf_locustfile.py	Adds --endpoint selection and renames the Locust user to a generic writer to support C++ + JS variants.
tests/basicperf_locust.py	Selects the appropriate endpoint variant based on --js-app-bundle and updates benchmark description.
CMakeLists.txt	Replaces the old JS Piccolo perf test with a new Locust-driven JS benchmark test target and parameters.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Description

Comparing 5 available runs from this branch (#8229) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 5 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#2DA44E!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 69,719 tx/s ▬ +1%"]
axis b2["Basic Blocking Locust 100ms: 3,057 tx/s ▬ 0%"]
axis b3["Basic Blocking Locust 20ms: 15,134 tx/s ▬ 0%"]
axis b4["Basic Blocking Locust 2ms: 30,971 tx/s ▬ -1%"]
axis b6["Basic JS Locust: 7,740 tx/s ▲ 3%"]
axis b8["Historical Queries: 203,641 tx/s ▬ -1%"]
axis b9["Logging: 66,046 tx/s ▬ +3%"]
axis b11["L…g JWT Blocking Locust: 14,903 tx/s ▲ 1%"]
curve stddev2_high["main EWMA + 2 std dev"]{107.99, 100.55, 100.66, 110.82, 103.12, 108.73, 108.76, 102.67}
curve stddev1_high["main EWMA + 1 std dev"]{104.00, 100.28, 100.33, 105.41, 101.56, 104.36, 104.38, 101.34}
curve stddev1_low["main EWMA - 1 std dev"]{96.00, 99.72, 99.67, 94.59, 98.44, 95.64, 95.62, 98.66}
curve stddev2_low["main EWMA - 2 std dev"]{92.01, 99.45, 99.34, 89.18, 96.88, 91.27, 91.24, 97.33}
curve branch_0["#8229 (4 runs earlier)"]{102.61, 100.32, 100.01, 100.95, 100.00, 98.47, 102.68, 100.00}
curve branch_1["#8229 (3 runs earlier)"]{101.44, 100.15, 100.08, 99.28, 100.97, 103.39, 104.14, 100.18}
curve branch_2["#8229 (2 runs earlier)"]{102.44, 100.26, 100.35, 103.56, 103.79, 103.10, 100.99, 100.47}
curve branch_3["#8229 (1 run earlier)"]{103.19, 100.13, 100.20, 106.77, 103.95, 105.81, 106.43, 102.05}
curve branch_4["#8229"]{100.91, 100.29, 99.96, 99.42, 102.73, 99.14, 102.84, 101.36}
graticule polygon
max 119
min 81
ticks 0
showLegend false

Loading

Latency (ms)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking Locust 100ms: 99 ms ▬ 0%"]
axis b1["Basic Blocking Locust 20ms: 19 ms ▬ 0%"]
axis b2["Basic Blocking Locust 2ms: 9 ms ▬ -1%"]
axis b3["Basic JS Locust: 39 ms ▼ 2%"]
axis b4["Commit Latency 16ms: 5.17 ms ▬ +5%"]
axis b5["Commit Latency 1ms: 1.93 ms ▬ 0%"]
axis b6["Commit Latency 256ms: 213 ms ▬ 0%"]
axis b7["Logging JWT Blocking Locust: 20 ms ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 108.91, 103.74, 147.05, 105.51, 102.37, 100.00}
curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 104.46, 101.87, 123.53, 102.76, 101.19, 100.00}
curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 95.54, 98.13, 76.47, 97.24, 98.81, 100.00}
curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 91.09, 96.26, 52.95, 94.49, 97.63, 100.00}
curve branch_0["#8229 (4 runs earlier)"]{100.00, 100.00, 98.58, 100.00, 101.63, 98.99, 100.58, 100.00}
curve branch_1["#8229 (3 runs earlier)"]{100.00, 100.00, 98.58, 97.50, 97.47, 109.57, 99.61, 100.00}
curve branch_2["#8229 (2 runs earlier)"]{100.00, 100.00, 98.58, 95.00, 95.14, 101.44, 100.53, 100.00}
curve branch_3["#8229 (1 run earlier)"]{100.00, 100.00, 87.62, 95.00, 83.05, 101.39, 100.51, 95.00}
curve branch_4["#8229"]{100.00, 100.00, 98.58, 97.50, 104.99, 100.29, 100.20, 100.00}
graticule polygon
max 180
min 20
ticks 0
showLegend false

Loading

Memory (bytes)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 84.6 MiB ▼ 4%"]
axis b2["Basic Blocking Locust 100ms: 91.4 MiB ▬ +1%"]
axis b3["Basic Blocking Locust 20ms: 91.7 MiB ▲ 1%"]
axis b4["Basic Blocking Locust 2ms: 93.5 MiB ▬ 0%"]
axis b6["Basic JS Locust: 93.4 MiB ▲ 1%"]
axis b8["Logging: 76.8 MiB ▬ 0%"]
axis b10["Logging JWT Blocking Locust: 91.3 MiB ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{105.52, 101.93, 101.57, 101.86, 101.45, 101.74, 102.35}
curve stddev1_high["main EWMA + 1 std dev"]{102.76, 100.97, 100.78, 100.93, 100.73, 100.87, 101.17}
curve stddev1_low["main EWMA - 1 std dev"]{97.24, 99.03, 99.22, 99.07, 99.27, 99.13, 98.83}
curve stddev2_low["main EWMA - 2 std dev"]{94.48, 98.07, 98.43, 98.14, 98.55, 98.26, 97.65}
curve branch_0["#8229 (4 runs earlier)"]{99.73, 98.66, 99.30, 99.99, 100.00, 98.72, 99.04}
curve branch_1["#8229 (3 runs earlier)"]{106.64, 100.27, 99.92, 99.45, 99.98, 100.41, 98.98}
curve branch_2["#8229 (2 runs earlier)"]{100.09, 99.54, 101.81, 98.85, 101.70, 99.27, 97.76}
curve branch_3["#8229 (1 run earlier)"]{103.20, 102.01, 101.05, 101.02, 99.94, 98.58, 98.95}
curve branch_4["#8229"]{96.43, 100.80, 101.10, 99.72, 101.12, 100.17, 99.80}
graticule polygon
max 111
min 90
ticks 0
showLegend false

Loading

Rate (ops/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["CHAMP get: 37,996,994 ops/s ▬ 0%"]
axis b1["CHAMP put: 5,443,898 ops/s ▬ 0%"]
axis b2["KV deserialisation: 1,650,710 ops/s ▬ +1%"]
axis b3["KV serialisation: 1,369,675 ops/s ▬ -1%"]
axis b4["KV s…t deserialisation: 4,161 ops/s ▬ 0%"]
axis b5["KV snapshot serialisation: 4,484 ops/s ▬ +2%"]
curve stddev2_high["main EWMA + 2 std dev"]{102.35, 103.37, 105.65, 106.93, 104.20, 111.72}
curve stddev1_high["main EWMA + 1 std dev"]{101.18, 101.69, 102.82, 103.46, 102.10, 105.86}
curve stddev1_low["main EWMA - 1 std dev"]{98.82, 98.31, 97.18, 96.54, 97.90, 94.14}
curve stddev2_low["main EWMA - 2 std dev"]{97.65, 96.63, 94.35, 93.07, 95.80, 88.28}
curve branch_0["#8229 (4 runs earlier)"]{99.49, 101.47, 101.27, 100.47, 101.57, 101.31}
curve branch_1["#8229 (3 runs earlier)"]{100.40, 100.59, 101.47, 101.19, 100.87, 97.63}
curve branch_2["#8229 (2 runs earlier)"]{100.64, 100.47, 102.65, 101.19, 102.26, 107.28}
curve branch_3["#8229 (1 run earlier)"]{100.12, 101.32, 100.29, 101.46, 102.34, 110.57}
curve branch_4["#8229"]{100.36, 99.57, 101.47, 98.82, 99.91, 102.10}
graticule polygon
max 120
min 80
ticks 0
showLegend false

Loading