Port historical query benchmark to Locust by achamayou · Pull Request #8230 · microsoft/CCF

Amaury Chamayou (achamayou) · 2026-08-27T21:28:29Z

Summary

replace the bespoke historical-query performance loop with a Locust workload, following the wrapper/workload split used in Replace Logging Piccolo benchmark with Locust #8206
pre-populate and warm historical state, validate every expected entry, then measure completed paginated range queries against a backup
preserve throughput reporting in entries/s while adding complete-query p50/p99 latency and backup memory metrics
extend the shared Locust runner to support a selected statistics row, workload-specific throughput units, and a non-primary target
publish the new workload under the distinct Historical Queries Locust Bencher label

Validation

ran Black, Ruff, gersemi, and the ASCII check on the changed files
compiled the changed Python files
loaded both benchmark CLIs with Locust and exercised argument discovery
exercised 202 retry, pagination, custom entry-count metrics, and entry-throughput parsing with focused Python checks
configured CMake with BUILD_TESTS=ON and confirmed the generated historical_query_locust CTest command
attempted to build the logging app; the local Clang 21 environment is blocked by QuickJS's version file shadowing the C++ <version> header

Measure completed historical range queries on a backup while preserving entry throughput reporting and validating every returned entry count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 729fed47-58cb-4c88-b7d6-c065c498b483

Copilot

Pull request overview

This PR ports the historical range-query performance benchmark from a bespoke Python loop to a Locust-driven workload, aligning with the shared Locust orchestration pattern already used by other CCF perf benchmarks. It adds query-level latency and backup memory reporting while preserving throughput reporting (now optionally in workload-specific units like entries/s).

Changes:

Extend the shared Locust benchmark runner to support selecting a specific statistics row, custom throughput units (including response-length-derived units), and targeting a non-primary node.
Add a new historical range-query Locust workload + benchmark driver that pre-populates/warm-ups historical state, validates expected counts, then measures paginated range queries against a backup.
Replace the prior historical_query_perf.py perf test wiring with the new Locust-based benchmark in CMake under a distinct Bencher label.

Custom instructions used:

.github/copilot-instructions.md

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/infra/locust_benchmark.py	Generalizes Locust orchestration/reporting to support workload-selected stats rows, throughput units, and non-primary targets.
tests/infra/historical_query_locustfile.py	Implements Locust workload that executes paginated historical range queries and emits query-level metrics (latency + entry count).
tests/historical_query_perf.py	Removes the previous bespoke historical query perf benchmark implementation.
tests/historical_query_locust.py	Adds the new benchmark driver that prepares/warm-ups data, validates counts, then runs Locust against a backup.
CMakeLists.txt	Rewires the perf test entry to run the new Locust benchmark with the new Bencher label and parameters.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-27T22:50:43Z

Description

Comparing 5 available runs from this branch (#8230) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 5 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 68,377 tx/s ▬ -2%"]
axis b1["Basic Blocking Locust 100ms: 3,041 tx/s ▬ 0%"]
axis b2["Basic Blocking Locust 20ms: 15,105 tx/s ▬ 0%"]
axis b3["Basic Blocking Locust 2ms: 31,062 tx/s ▬ -1%"]
axis b5["Basic JS Locust: 7,344 tx/s ▼ 5%"]
axis b8["H…l Queries Locust: 552,905 tx/s ▼ 10%"]
axis b10["L…g C…e Blocking Locust: 23,510 tx/s ▬ -1%"]
axis b12["L…g JWT Blocking Locust: 14,786 tx/s ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{106.30, 100.45, 100.54, 109.68, 105.10, 109.47, 106.06, 102.62}
curve stddev1_high["main EWMA + 1 std dev"]{103.15, 100.23, 100.27, 104.84, 102.55, 104.74, 103.03, 101.31}
curve stddev1_low["main EWMA - 1 std dev"]{96.85, 99.77, 99.73, 95.16, 97.45, 95.26, 96.97, 98.69}
curve stddev2_low["main EWMA - 2 std dev"]{93.70, 99.55, 99.46, 90.32, 94.90, 90.53, 93.94, 97.38}
curve branch_4["#8230"]{98.04, 99.70, 99.67, 98.70, 95.47, 89.54, 99.29, 100.10}
graticule polygon
max 117
min 82
ticks 0
showLegend false

Latency (ms)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(9){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(10){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking Locust 100ms: 99 ms ▬ 0%"]
axis b1["Basic Blocking Locust 20ms: 19 ms ▬ 0%"]
axis b2["Basic Blocking Locust 2ms: 9 ms ▬ 0%"]
axis b3["Basic JS Locust: 40 ms ▲ 5%"]
axis b4["Commit Latency 16ms: 4.6 ms ▬ -7%"]
axis b5["Commit Latency 1ms: 1.87 ms ▼ 4%"]
axis b6["Commit Latency 256ms: 214 ms ▬ 0%"]
axis b7["Historical Queries Locust: 56 ms ▲ 10%"]
axis b8["L…g Certificate Blocking Locust: 21 ms ▼ 5%"]
axis b9["Logging JWT Blocking Locust: 20 ms ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 109.08, 105.24, 135.20, 105.36, 101.68, 109.80, 106.43, 100.00}
curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 104.54, 102.62, 117.60, 102.68, 100.84, 104.90, 103.21, 100.00}
curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 95.46, 97.38, 82.40, 97.32, 99.16, 95.10, 96.79, 100.00}
curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 90.92, 94.76, 64.80, 94.64, 98.32, 90.20, 93.57, 100.00}
curve branch_4["#8230"]{100.00, 100.00, 100.40, 104.74, 93.21, 96.19, 100.22, 109.80, 95.42, 100.00}
graticule polygon
max 160
min 40
ticks 0
showLegend false

Memory (bytes)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 84.7 MiB ▬ -3%"]
axis b1["Basic Blocking Locust 100ms: 91 MiB ▬ +1%"]
axis b2["Basic Blocking Locust 20ms: 91 MiB ▬ 0%"]
axis b3["Basic Blocking Locust 2ms: 93.1 MiB ▬ -1%"]
axis b5["Basic JS Locust: 93.3 MiB ▲ 1%"]
axis b7["Historical Queries Locust: 149 MiB ▼ 1%"]
axis b9["L…g C…e Blocking Locust: 119 MiB ▬ +1%"]
axis b11["Logging JWT Blocking Locust: 90.7 MiB ▬ -1%"]
curve stddev2_high["main EWMA + 2 std dev"]{105.66, 101.89, 101.44, 101.86, 100.02, 100.73, 102.62, 102.75}
curve stddev1_high["main EWMA + 1 std dev"]{102.83, 100.95, 100.72, 100.93, 100.01, 100.37, 101.31, 101.38}
curve stddev1_low["main EWMA - 1 std dev"]{97.17, 99.05, 99.28, 99.07, 99.99, 99.63, 98.69, 98.62}
curve stddev2_low["main EWMA - 2 std dev"]{94.34, 98.11, 98.56, 98.14, 99.98, 99.27, 97.38, 97.25}
curve branch_4["#8230"]{97.41, 100.52, 100.41, 99.24, 101.01, 99.27, 101.13, 99.15}
graticule polygon
max 110
min 90
ticks 0
showLegend false

Rate (ops/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(2){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
cScale8: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["CHAMP get: 38,576,730 ops/s ▲ 2%"]
axis b1["CHAMP put: 5,364,402 ops/s ▼ 2%"]
axis b2["KV deserialisation: 1,560,062 ops/s ▼ 5%"]
axis b3["KV serialisation: 1,384,850 ops/s ▬ -1%"]
axis b4["KV s…t deserialisation: 4,174 ops/s ▬ -1%"]
axis b5["KV snapshot serialisation: 4,062 ops/s ▼ 8%"]
curve stddev2_high["main EWMA + 2 std dev"]{101.68, 103.18, 103.12, 104.83, 102.11, 110.33}
curve stddev1_high["main EWMA + 1 std dev"]{100.84, 101.59, 101.56, 102.42, 101.05, 105.17}
curve stddev1_low["main EWMA - 1 std dev"]{99.16, 98.41, 98.44, 97.58, 98.95, 94.83}
curve stddev2_low["main EWMA - 2 std dev"]{98.32, 96.82, 96.88, 95.17, 97.89, 89.67}
curve branch_0["#8230 (4 runs earlier)"]{100.12, 101.03, 100.19, 99.75, 101.12, 105.64}
curve branch_1["#8230 (3 runs earlier)"]{101.10, 99.14, 102.39, 100.73, 99.69, 106.48}
curve branch_2["#8230 (2 runs earlier)"]{99.85, 99.89, 100.34, 99.60, 100.06, 97.88}
curve branch_3["#8230 (1 run earlier)"]{100.98, 100.47, 100.01, 100.73, 99.41, 93.00}
curve branch_4["#8230"]{101.94, 98.12, 94.85, 99.04, 99.50, 91.86}
graticule polygon
max 118
min 82
ticks 0
showLegend false

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 729fed47-58cb-4c88-b7d6-c065c498b483

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Amaury Chamayou (achamayou) requested a review from a team as a code owner August 27, 2026 21:28

CopilotAI lite review requested due to automatic review settings August 27, 2026 21:28

Copilotstarted reviewing on behalf of Amaury Chamayou (achamayou)August 27, 2026 21:29 View session

CopilotAI reviewed Aug 27, 2026
View reviewed changes

Comment threadtests/infra/locust_benchmark.py

Amaury Chamayou (achamayou) added the bench-ab label Aug 27, 2026

Amaury Chamayou (achamayou)and others added 2 commits August 28, 2026 07:53

Report missing Locust statistics columns clearly
f879671
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 729fed47-58cb-4c88-b7d6-c065c498b483

Merge branch 'main' into achamayou-historical-query-locust
5553c55

Eddy Ashton (eddyashton) approved these changes Aug 28, 2026
View reviewed changes

Amaury Chamayou (achamayou) added 2 commits August 28, 2026 14:08

Merge branch 'main' into achamayou-historical-query-locust
4572f46

Merge branch 'main' into achamayou-historical-query-locust
107dbff

Amaury Chamayou (achamayou) enabled auto-merge (squash) August 28, 2026 13:41

Amaury Chamayou (achamayou)and others added 2 commits August 28, 2026 15:22

Merge branch 'main' into achamayou-historical-query-locust
faf71f5
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into achamayou-historical-query-locust
e44274c

Amaury Chamayou (achamayou) merged commit b40dbce into mainAug 28, 2026
12 checks passed

Amaury Chamayou (achamayou) deleted the achamayou-historical-query-locust branch August 28, 2026 15:14

Navigation Menu

Uh oh!

Port historical query benchmark to Locust - #8230

Port historical query benchmark to Locust#8230
Amaury Chamayou (achamayou) merged 7 commits into
mainfrom
achamayou-historical-query-locust

Amaury Chamayou (achamayou) commented Aug 27, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

Uh oh!

github-actionsBot commented Aug 27, 2026•
edited
Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Amaury Chamayou (achamayou) commented Aug 27, 2026

Summary

Validation

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actionsBot commented Aug 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Throughput (tx/s)

Latency (ms)

Memory (bytes)

Rate (ops/s)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actionsBot commented Aug 27, 2026•
edited
Loading