Skip to content

Benchmark Locust logging with RPCConnectionManager - #8216

Closed
Amaury Chamayou (achamayou) wants to merge 79 commits into
achamayou-logging-locust-benchmarkfrom
achamayou-rpc-manager-benchmark
Closed

Benchmark Locust logging with RPCConnectionManager#8216
Amaury Chamayou (achamayou) wants to merge 79 commits into
achamayou-logging-locust-benchmarkfrom
achamayou-rpc-manager-benchmark

Conversation

@achamayou

@achamayouAmaury Chamayou (achamayou) commented Aug 27, 2026

Copy link
Copy Markdown
Member

Summary

Applies the exact #8117 head (b3990627387a45cf1f41a5864b51fc30c3c53db2) on top of the exact #8206 head (9c95e9a60eb087ca5bbdd12d2dec271e99c270df) solely to collect combined Locust logging benchmark numbers.

Depends on #8206 and #8117.

Conflict resolution

Validation

  • Built ccf_launcher, task_system_test, and openssl_server_test.
  • task_system_test: 15 cases, 290 assertions passed.
  • openssl_server_test: 31 cases, 585 assertions passed.
  • C++ formatting, release-note validation, staged diff checks, and ASCII checks passed.

A full js_generic build was not usable in this Windows-backed WSL worktree because Git symlink files under QuickJS are materialized as regular files; the focused host launcher and unit-test targets above compile successfully.

Worker and client-count comparison

The two-worker sweep and three-worker sweep ran the certificate-authenticated blocking Locust workload at the same eight client counts used for #8206.

RPCConnectionManager two-worker throughput and latency

Clients2 workers3 workersChangep50 (2W/3W)p99 (2W/3W)
32014,990.0 tx/s15,134.9 tx/s+1.0%19/19 ms23/22 ms
48021,035.5 tx/s21,830.3 tx/s+3.8%19/18 ms33/24 ms
56022,793.3 tx/s24,388.5 tx/s+7.0%19/19 ms38/36 ms
64021,780.2 tx/s26,094.8 tx/s+19.8%21/20 ms43/38 ms
72020,373.2 tx/s25,245.2 tx/s+23.9%25/22 ms47/43 ms
80019,473.8 tx/s23,226.0 tx/s+19.3%33/25 ms52/46 ms
90020,021.4 tx/s22,628.3 tx/s+13.0%36/34 ms62/52 ms
1,00019,892.9 tx/s22,379.0 tx/s+12.5%39/37 ms71/60 ms

Unlike the original transport results on #8206, where a third worker did not raise the approximately 25k tx/s ceiling, RPCConnectionManager benefits materially from the third worker at saturation. The two-worker curve peaks at 560 clients at 22.8k tx/s; the three-worker curve peaks at 640 clients at 26.1k tx/s. Below saturation the difference is small, but from 640 clients onward the third worker improves throughput by 12.5-23.9% and lowers tail latency.

The branch is restored to the normal 2-worker, 640-client configuration. Final restored run 33065616806 passed at 24,797.6 tx/s, 20ms p50, 40ms p99, and 112.0 MiB.

…ransport cert-deferred listening + ALPN + outbound client, RPCConnectionManager (AbstractRPCSessions). Not yet wired into enclave/run.cpp.
…wire RPCConnectionManager into enclave.h/run.cpp, delete RPCSessions/rpc_connections/tls_session. Full build green, 53/53 unit tests pass.
…:Cert::use) and request client cert on inbound for caller auth; add peer-cert capture test. Full build green, unit tests pass.
…xing localhost/[::1] interfaces (cpp, cpp_cose_only, common_ipv6 e2e). Add localhost/IPv6 binding tests.
… large response queued just before close_socket() is not truncated (fixes cpp/cpp_cose_only receipt 'server disconnected'). Add truncation test.
…e (ERR_clear_error) before each SSL op so a stale error from one connection cannot poison SSL_get_error for another (root cause of cpp/cpp_cose_only 'server disconnected'). Add persistent-connection + peer-cert tests.
… handler; branch udp interfaces to listen_udp. Clearly marked QUIC extension points (substrate for OpenSSL >=3.5 native QUIC). e2e_logging udp echo passes; full suite green.
…ned quic_session.h/src/quic, udp.h + udp/msg_types.h + UDPImpl vestiges in run.cpp, dead RPC ringbuffer message enums (keep tcp::ConnID). Drop old-implementation comments. Fix build after cert.h use->configure_ssl rename + commit-callback include.
…Config (so the enclave-side RPC manager receives it); track per-connection last_active in OpenSSLServer and sweep idle connections off the epoll timeout. Plumbed manager->bridge->server. idletimeout e2e passes.
Notify the enclave work beacon when transport tasks enter the JobBoard, while preserving direct worker handoff and coalescing redundant wakeups. Keep bounded task drains moving immediately when a backlog remains.\n\nRefs #8117\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…code
- Modify CMakeLists.txt for better configuration.
- Refactor rpc_tls_client.h for improved clarity and functionality.
- Enhance openssl_server_test.cpp with additional test cases.
RPC task workers can query consensus state concurrently with Raft message processing. Publish a coherent query snapshot without taking the Raft lock from KV-backed endpoints, avoiding both data races and KV/Raft lock inversion.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts:
#	CMakeLists.txt
#	src/tls/test/main.cpp
These changes were added while chasing TSAN failures exposed during the RPC connection manager work, but they are broader Raft synchronization changes and should be handled separately rather than bundled into this PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Amaury Chamayou <amchamay@microsoft.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actionsBot commented Aug 27, 2026

Copy link
Copy Markdown
Description

Comparing 4 available runs from this branch (#8216) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 4 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-5{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(7){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(8){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(9){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 78,262 tx/s ▲ 11%"]
axis b2["Basic Blocking Locust 100ms: 3,052 tx/s ▬ 0%"]
axis b3["Basic Blocking Locust 20ms: 15,126 tx/s ▬ 0%"]
axis b4["Basic Blocking Locust 2ms: 25,416 tx/s ▼ 21%"]
axis b5["Basic JS: 5,023 tx/s ▲ 6%"]
axis b6["Basic Multi-Threaded: 128,482 tx/s ▲ 48%"]
axis b7["Historical Queries: 234,922 tx/s ▲ 13%"]
axis b9["L…g C…e Blocking Locust: 24,798 tx/s ▲ 3%"]
axis b11["L…g JWT Blocking Locust: 14,772 tx/s ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{107.51, 100.59, 100.68, 109.19, 107.14, 106.81, 111.50, 103.49, 101.60}
curve stddev1_high["main EWMA + 1 std dev"]{103.76, 100.29, 100.34, 104.60, 103.57, 103.40, 105.75, 101.74, 100.80}
curve stddev1_low["main EWMA - 1 std dev"]{96.24, 99.71, 99.66, 95.40, 96.43, 96.60, 94.25, 98.26, 99.20}
curve stddev2_low["main EWMA - 2 std dev"]{92.49, 99.41, 99.32, 90.81, 92.86, 93.19, 88.50, 96.51, 98.40}
curve branch_0["#8216 (3 runs earlier)"]{106.98, 99.62, 98.85, 79.49, 104.58, 143.14, 114.23, 100.00, 100.55}
curve branch_3["#8216"]{110.86, 99.92, 99.80, 79.36, 105.92, 147.97, 112.51, 103.49, 100.46}
graticule polygon
max 172
min 55
ticks 0
showLegend false
Loading

Latency (ms)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-5{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(2){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#2DA44E!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking Locust 100ms: 98 ms ▼ 1%"]
axis b1["Basic Blocking Locust 20ms: 18 ms ▼ 5%"]
axis b2["Basic Blocking Locust 2ms: 11 ms ▲ 22%"]
axis b3["Commit Latency 16ms: 4.07 ms ▬ -12%"]
axis b4["Commit Latency 1ms: 1.82 ms ▼ 7%"]
axis b5["Commit Latency 256ms: 205 ms ▼ 4%"]
axis b6["L…g Certificate Blocking Locust: 20 ms ▬ 0%"]
axis b7["Logging JWT Blocking Locust: 19 ms ▼ 5%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 108.27, 152.75, 104.41, 102.84, 100.00, 100.00}
curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 104.13, 126.38, 102.21, 101.42, 100.00, 100.00}
curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 95.87, 73.62, 97.79, 98.58, 100.00, 100.00}
curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 91.73, 47.25, 95.59, 97.16, 100.00, 100.00}
curve branch_0["#8216 (3 runs earlier)"]{98.99, 94.74, 122.46, 116.03, 88.28, 95.23, 100.00, 95.00}
curve branch_3["#8216"]{98.99, 94.74, 122.46, 87.67, 93.23, 96.03, 100.00, 95.00}
graticule polygon
max 190
min 10
ticks 0
showLegend false
Loading

Memory (bytes)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-5{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(2){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(3){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(4){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#2DA44E!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic: 92 MiB ▲ 4%"]
axis b2["Basic Blocking Locust 100ms: 85.6 MiB ▼ 5%"]
axis b3["Basic Blocking Locust 20ms: 87.4 MiB ▼ 4%"]
axis b4["Basic Blocking Locust 2ms: 88.6 MiB ▼ 5%"]
axis b5["Basic JS: 74.8 MiB ▲ 6%"]
axis b6["Basic Multi-Threaded: 92 MiB ▲ 3%"]
axis b8["L…g C…e Blocking Locust: 112 MiB ▬ 0%"]
axis b10["Logging JWT Blocking Locust: 84.7 MiB ▼ 8%"]
curve stddev2_high["main EWMA + 2 std dev"]{104.24, 102.15, 101.64, 101.83, 103.83, 102.57, 100.26, 102.24}
curve stddev1_high["main EWMA + 1 std dev"]{102.12, 101.07, 100.82, 100.91, 101.91, 101.29, 100.13, 101.12}
curve stddev1_low["main EWMA - 1 std dev"]{97.88, 98.93, 99.18, 99.09, 98.09, 98.71, 99.87, 98.88}
curve stddev2_low["main EWMA - 2 std dev"]{95.76, 97.85, 98.36, 98.17, 96.17, 97.43, 99.74, 97.76}
curve branch_0["#8216 (3 runs earlier)"]{104.78, 95.67, 98.64, 94.71, 104.77, 104.87, 100.00, 94.12}
curve branch_3["#8216"]{104.27, 94.55, 96.20, 94.66, 105.77, 103.11, 100.26, 91.86}
graticule polygon
max 111
min 86
ticks 0
showLegend false
Loading

Rate (ops/s)

---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.30!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-7{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
cScale7: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["CHAMP get: 38,023,802 ops/s ▬ 0%"]
axis b1["CHAMP put: 5,557,727 ops/s ▬ +2%"]
axis b2["KV deserialisation: 1,639,613 ops/s ▬ +1%"]
axis b3["KV serialisation: 1,438,849 ops/s ▬ 0%"]
axis b4["KV s…t deserialisation: 4,176 ops/s ▬ 0%"]
axis b5["KV snapshot serialisation: 4,600 ops/s ▬ +3%"]
curve stddev2_high["main EWMA + 2 std dev"]{105.58, 106.40, 106.20, 107.16, 105.65, 111.89}
curve stddev1_high["main EWMA + 1 std dev"]{102.79, 103.20, 103.10, 103.58, 102.83, 105.94}
curve stddev1_low["main EWMA - 1 std dev"]{97.21, 96.80, 96.90, 96.42, 97.17, 94.06}
curve stddev2_low["main EWMA - 2 std dev"]{94.42, 93.60, 93.80, 92.84, 94.35, 88.11}
curve branch_0["#8216 (3 runs earlier)"]{100.38, 100.39, 99.72, 100.24, 98.79, 102.60}
curve branch_1["#8216 (2 runs earlier)"]{100.40, 100.71, 97.52, 90.69, 99.06, 93.08}
curve branch_2["#8216 (1 run earlier)"]{100.38, 100.27, 102.38, 97.18, 102.89, 103.74}
curve branch_3["#8216"]{100.44, 102.29, 101.04, 100.11, 100.49, 103.39}
graticule polygon
max 121
min 79
ticks 0
showLegend false
Loading

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6435b6e1-97c1-49da-bb1d-59d0ae08364f
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6435b6e1-97c1-49da-bb1d-59d0ae08364f
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6435b6e1-97c1-49da-bb1d-59d0ae08364f
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@achamayou@eddyashton