Skip to content

Run benchmarks and configure Grafana dashboard - #5

Merged
viqueen merged 23 commits into
mainfrom
feature/run-benchmark-and-configure-grafana
Mar 8, 2026
Merged

Run benchmarks and configure Grafana dashboard#5
viqueen merged 23 commits into
mainfrom
feature/run-benchmark-and-configure-grafana

Conversation

@viqueen

@viqueenviqueen commented Mar 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Run k6 load tests via Docker (grafana/k6:1.6.1) with gRPC reflection mode support
  • Fix benchmark result collection: exec return values, gRPC metric parsing, version detection
  • Replace custom OTLP publisher with OpenTelemetry JS SDK, using standard OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS env vars
  • Add .env support for Grafana Cloud credentials with dotenv
  • Add importable Grafana dashboard (grafana/benchmark-dashboard.json) with overview stats, latency comparison, throughput, and summary table
  • Add build/fix devloop scripts to toolkit and root
  • Pin Docker image versions, support standalone publish command for multiple result files

Test plan

  • Run npm run build to verify lint/format passes
  • Run npm run benchmark -- run content-api/connect-rpc to execute a full benchmark
  • Run npm run benchmark -- publish results/content-api-connect-rpc-*.json to publish results
  • Import grafana/benchmark-dashboard.json into Grafana Cloud and verify panels populate

🤖 Generated with Claude Code

viqueenand others added 14 commits March 8, 2026 09:22
@viqueen@claude
Run k6 via Docker and remove remote import in k6 script
bfcbefc
Run k6 load tests using grafana/k6 Docker image instead of requiring
a local k6 binary. Mounts workspace at /workspace (read-only) and
remaps file paths (PROTO_DIR etc.) to container paths. Uses
--network host for target access.
Replace remote randomString import with built-in crypto.randomUUID()
in the k6 script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Enable gRPC reflection and use k6 reflect mode
0d7d58f
Add connectrpc.com/grpcreflect to the server so k6 can discover
services at runtime via reflect:true. Remove proto file loading
from the k6 script and PROTO_DIR from benchmark config. Update
agent and scaffold command accordingly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Fix k6 FieldMask serialization for gRPC reflection mode
4879a6c
Use comma-separated string format for google.protobuf.FieldMask as
required by protobuf JSON encoding spec, fixing serialization error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Fix toolkit result collection: exec return, gRPC metrics, versions
ee6e435
- Return execa result from exec() so version collection works
- Parse gRPC metrics (grpc_req_duration) alongside HTTP metrics
- Remove .values wrapper missing in k6 summary-export format
- Use protocol-agnostic field names (reqs, reqDuration, reqFailedRate)
- Get k6 version from Docker image instead of local binary
- Update all consumers (compare, formatter, run, loadtest commands)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Add .env support for Grafana credentials and update README
4bfd990
- Add dotenv to auto-load .env file on CLI startup
- Create .env.example with Grafana Cloud variables
- Update README: add Grafana setup section, remove local k6 prerequisite,
update metric names to protocol-agnostic format
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Pin grafana/k6 Docker image to v1.6.1
8fa4766
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Clarify Grafana Cloud credential instructions in .env.example and README
d942e65
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Simplify Grafana config to endpoint + token
3f3a0cb
Replace instanceId/apiKey with a single base64 token that Grafana Cloud
generates directly on the OTLP configuration page. Also make the
endpoint configurable via env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Use standard OTel env vars for Grafana Cloud credentials
5e5b240
Align on OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS
which are the exact env vars Grafana Cloud generates on the OTLP
configuration page. Parse the headers string at publish time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Support multiple files in publish command
a1d1048
Accept variadic arguments so results can be batch-published without
needing to re-run benchmarks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Replace custom OTLP publisher with OpenTelemetry JS SDK
49f2b8e
Use @opentelemetry/exporter-metrics-otlp-http which natively reads
OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS from env
vars. Remove custom header parsing, OTLP formatter, and grafana config
section from benchmark.config.json.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Update README quick start with publish command, clear .env.example va…
057fa18
…lues
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Fix OTel v2 imports, add devloop scripts, move deps to toolkit
df455fb
- Fix resourceFromAttributes import (OTel resources v2 API change)
- Use static import for PeriodicExportingMetricReader
- Add build/fix scripts to toolkit and root package.json
- Move OTel and dotenv deps from root to toolkit workspace
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Add Grafana dashboard and fix OTLP metric visibility
4a45aa1
- Add importable Grafana dashboard (grafana/benchmark-dashboard.json)
with overview stats, latency comparison, throughput, and summary table
- Move benchmark attributes from resource to data point attributes so
they appear as Prometheus labels without requiring promotion
- Remove metric units to avoid Grafana Cloud appending suffixes
- Update README with dashboard import instructions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings March 8, 2026 00:22
@viqueen@claude
Fix stat panels to use instant queries for gauge data
4787b20
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CopilotAI reviewed Mar 8, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the benchmark toolkit to run k6 load tests via a pinned Docker image, publish results to Grafana Cloud using the OpenTelemetry JS SDK and standard OTEL env vars, and adds an importable Grafana dashboard for comparing benchmark runs.

Changes:

  • Run k6 inside Docker (grafana/k6:1.6.1) and enable gRPC reflection-based scripts (no local k6 / proto load required)
  • Replace the custom Grafana Cloud OTLP publisher with OpenTelemetry JS OTLP/HTTP metrics export + .env loading
  • Add a Grafana dashboard JSON and update CLI/docs/config accordingly (including multi-file publish)

Reviewed changes

Copilot reviewed 23 out of 26 changed files in this pull request and generated 8 comments.

Show a summary per file
FileDescription
toolkit/src/util/exec.jsReturn execa() result so callers can read stdout/stderr (needed for version/result collection).
toolkit/src/report/compare.jsAlign comparison output fields with the new parsed k6 summary shape.
toolkit/src/publish/otlp.jsNew OpenTelemetry-based OTLP metrics publisher using standard OTEL_EXPORTER_OTLP_* env vars.
toolkit/src/publish/grafana-cloud.jsRemoved custom Grafana Cloud client in favor of OTel SDK.
toolkit/src/publish/formatter.jsRemoved custom OTLP JSON formatting in favor of OTel SDK.
toolkit/src/metrics/k6-parser.jsUpdate k6 summary parsing to support HTTP vs gRPC metrics and renamed fields.
toolkit/src/metrics/collector.jsCollect k6 version via Docker image rather than local k6 binary.
toolkit/src/core/k6.jsRun k6 via Docker with mounted workspace/results; map env paths into container.
toolkit/src/config/schema.jsRemove Grafana config from the benchmark config schema.
toolkit/src/cli/commands/run.jsUpdate logging to new summary field names and publish via new OTLP exporter.
toolkit/src/cli/commands/publish.jsSupport publishing multiple result files; publish via new OTLP exporter (no config load).
toolkit/src/cli/commands/loadtest.jsUpdate logging to new summary field names.
toolkit/package.jsonAdd OTel metric exporter deps and devloop scripts (build, fix).
toolkit/bin/benchmark.jsLoad .env automatically via dotenv/config.
projects/content-api/connect-rpc/go.modAdd connectrpc.com/grpcreflect (and adjust direct/indirect deps).
projects/content-api/connect-rpc/go.sumAdd checksums for new grpcreflect dependency.
projects/content-api/connect-rpc/cmd/server/setup_gateway.goEnable gRPC reflection endpoints for tooling/k6 reflection mode.
projects/content-api/_shared/protobuf/k6/content-api.jsSwitch to reflect: true and remove manual proto loading.
package.jsonAdd root fix script that runs workspace fixes.
package-lock.jsonLockfile updates for new OTel dependencies and dotenv workspace placement.
grafana/benchmark-dashboard.jsonAdd importable Grafana dashboard for benchmark overview/latency/throughput comparisons.
benchmark.config.jsonRemove gRPC PROTO_DIR env since scripts use reflection now; remove grafana config block.
README.mdDocument Docker-based k6, OTEL env var publishing, dashboard import, and updated commands.
.env.exampleAdd example OTEL exporter env vars for Grafana Cloud.
.claude/commands/scaffold-implementation.mdUpdate scaffolding guidance to remove PROTO_DIR for gRPC targets.
.claude/agents/connect-rpc-go.mdUpdate agent docs/snippets to include gRPC reflection wiring.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadtoolkit/src/core/k6.js
Comment threadtoolkit/src/metrics/collector.js Outdated
Comment threadprojects/content-api/connect-rpc/cmd/server/setup_gateway.go
Comment threadREADME.md Outdated
Comment threadgrafana/benchmark-dashboard.json Outdated
Comment threadgrafana/benchmark-dashboard.json Outdated
Comment threadtoolkit/src/metrics/k6-parser.js
Comment threadtoolkit/src/metrics/k6-parser.js
viqueenand others added 8 commits March 8, 2026 12:42
@viqueen@claude
Address PR review comments
05ef7ce
- Fix k6 Docker networking: use --network host on Linux, --add-host
host.docker.internal on macOS/Windows for Docker Desktop compatibility
- Use k6 image tag as version instead of running a container to detect it
- Restore .values accessor in k6 summary parser (--summary-export format)
- Fix checksPassRate to use checks.values.rate instead of checks.value
- Fix dashboard panel descriptions: "in seconds" → "in milliseconds"
- Update README: clarify OTLP credentials are read at publish time
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Add per-stage Docker build metrics
2b5a404
- Parse BuildKit --progress=plain output to extract per-stage durations
(e.g. generate, builder, runtime)
- Publish per-stage metrics as benchmark.build.stage.duration with
benchmark.build.stage label for Grafana filtering
- Add stage columns to comparison table
- Add Build Stage Breakdown panel to Grafana dashboard
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Fix k6 connectivity on macOS and add health check before loadtest
754c9e6
- Restore --network host on all platforms (works on Docker Desktop)
- Rewrite localhost to host.docker.internal in k6 env vars on non-Linux
so k6 container can reach services published on the host
- Add health check to standalone loadtest command to prevent running
k6 against a service that is not ready
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Simplify Grafana dashboard for legibility
24241ac
Reduce from 12 panels to 6 by removing redundant stat panels,
duplicate bar gauges, and the summary table. Use consistent
bar gauge and bar chart panels with clear labels and
palette-classic-by-name coloring for easy target comparison.
Layout:
- Performance: Throughput + Error Rate (bar gauges)
- Latency: percentile distribution (bar chart)
- Build & Deploy: build time + deploy time (bar gauges)
- Build Stages: per-stage breakdown (bar chart)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Fix build stage parsing to check both stdout and stderr
ae3fcf0
BuildKit progress output may go to stdout or stderr depending on
Docker version and compose configuration. Parse both to reliably
extract per-stage durations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Replace custom build stage parsing with BuildKit OTLP traces
51ec813
BuildKit natively exports per-stage build traces via the same OTEL env
vars already configured for metrics publishing. This removes the custom
--progress=plain output parser which was unreliable, and delegates
build stage visibility to Grafana Cloud Tempo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Fix k6 summary parser to match k6 1.6.1 export format
146e348
k6 1.6.1 --summary-export puts metrics directly on the metric object
(e.g. metrics.grpc_req_duration.avg) without a .values wrapper. Also
fix checksPassRate to read checks.value instead of checks.rate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen@claude
Simplify Grafana dashboard layout
68db4ec
Remove row separator panels and tighten grid positions for a cleaner
5-panel layout.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@viqueen
viqueen enabled auto-merge (squash) March 8, 2026 03:14
@viqueen
viqueen merged commit 77e0d61 into mainMar 8, 2026
2 checks passed
@viqueen
viqueen deleted the feature/run-benchmark-and-configure-grafana branch March 8, 2026 03:16
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@viqueen