Skip to content

Add more Raft status metrics for Prometheus and Grafana dashboard - #386

Merged
bootjp merged 5 commits into
mainfrom
copilot/add-raft-status-metrics
Mar 21, 2026
Merged

Add more Raft status metrics for Prometheus and Grafana dashboard#386
bootjp merged 5 commits into
mainfrom
copilot/add-raft-status-metrics

Conversation

CopilotAI commented Mar 21, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Expands the existing Prometheus metrics exporter with additional Raft status gauges sourced from the already-polled raft.Raft.Stats() map — no extra RPC calls.

New metrics

MetricDescription
elastickv_raft_termCurrent Raft term — increments on each leader election
elastickv_raft_last_log_indexLast written log entry index — useful for cross-node replication lag
elastickv_raft_last_snapshot_indexMost recent snapshot index
elastickv_raft_fsm_pendingCommands queued to FSM but not yet applied — backpressure indicator
elastickv_raft_num_peersOther voting members in the cluster, excluding this node

All metrics carry the same group, node_id, and node_address labels as existing Raft gauges.

Grafana dashboard

A new dashboard monitoring/grafana/dashboards/elastickv-raft-status.json is included for visualising Raft health at a glance:

RowPanels
Cluster OverviewRaft Term · Voting Peers · Member Count · FSM Pending (stat, color-coded)
Leader & StateCurrent Leader per Group · Node Local State · Cluster Members (tables)
Index Progresslast log / commit / applied / snapshot index over time (time series)
Replication HealthReplication Lag (last_log_index − applied_index) · Last Leader Contact in seconds (time series)

The dashboard includes a group template variable for filtering by Raft group and auto-refreshes every 10 seconds.

Original prompt

This section details on the original issue you should resolve

<issue_title>Add metrics output for prometheus</issue_title>
<issue_description></issue_description>

<agent_instructions>add more raft status metrics</agent_instructions>

Comments on the Issue (you are @copilot in this section)

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

CopilotAI changed the title [WIP] Add metrics output for prometheusAdd more Raft status metrics for PrometheusMar 21, 2026
CopilotAI requested a review from bootjpMarch 21, 2026 09:57
@bootjp

bootjp commented Mar 21, 2026

Copy link
Copy Markdown
Owner

@copilot fix golangci-lint and ai require document add

CopilotAI commented Mar 21, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot fix golangci-lint and ai require document add

Fixed in 71eedf1:

  • golangci-lint: Removed extra alignment spaces from the RaftMetrics struct fields (the gci formatter requires standard gofmt alignment, not manual column alignment).
  • Doc comments: Added a type-level doc comment to RaftMetrics and per-field comments describing each gauge's purpose, including all five new fields (term, lastLogIndex, lastSnapshotIndex, fsmPending, numPeers).

golangci-lint run ./monitoring/... now reports 0 issues.

CopilotAI requested a review from bootjpMarch 21, 2026 10:09
@bootjp

bootjp commented Mar 21, 2026

Copy link
Copy Markdown
Owner

@copilot Raftの状態を確認するGrafanaダッシュボードのJSONを作ってください

CopilotAI commented Mar 21, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot Raftの状態を確認するGrafanaダッシュボードのJSONを作ってください

Added monitoring/grafana/dashboards/elastickv-raft-status.json in commit 7755874.

The dashboard has 4 rows:

RowPanels
Cluster OverviewRaft Term · Voting Peers · Member Count · FSM Pending (stat, color-coded)
Leader & StateCurrent Leader per Group · Node Local State · Cluster Members (tables)
Index Progresslast log / commit / applied / snapshot index over time (time series)
Replication HealthReplication Lag (last log − applied entries) · Last Leader Contact in seconds (time series)

A group template variable is included so you can filter by Raft group, and the dashboard auto-refreshes every 10 seconds.

CopilotAI changed the title Add more Raft status metrics for PrometheusAdd more Raft status metrics for Prometheus and Grafana dashboardMar 21, 2026
@bootjp
bootjp marked this pull request as ready for review March 21, 2026 11:08
@bootjp
bootjp enabled auto-merge March 21, 2026 11:08
@bootjp
bootjp merged commit ffa29f1 into mainMar 21, 2026
7 checks passed
@bootjp
bootjp deleted the copilot/add-raft-status-metrics branch March 21, 2026 11:10
bootjp added a commit that referenced this pull request Apr 30, 2026
@bootjp
backup: address review on S3 encoder (PR#718)
19ae328
Three correctness bugs and several quality issues, all valid.
CRITICAL: path traversal (Codex P1 #425).
S3 object keys are user-controlled. A key like "../../../etc/passwd"
would, under the prior filepath.Join-and-write code, escape the
bucket directory and overwrite host files. Added safeJoinUnderRoot
which Cleans the joined path and asserts it stays rooted under the
bucket dir; ".." traversal is rejected with ErrS3MalformedKey.
Absolute paths (leading "/") are normalised under the bucket dir by
filepath.Join, which is the safest outcome. Tests:
TestS3_PathTraversalAttemptRejected,
TestS3_AbsolutePathObjectKeyConfinedUnderBucket.
CRITICAL: stale upload-id chunks merged into body (Codex P1 #500,
Gemini HIGH #106/#476/#504).
A snapshot mid-delete-and-recreate or mid-retry can carry blob
chunks for multiple upload attempts under the same (bucket, gen,
object). The prior assembleObjectBody concatenated every chunk
regardless of upload_id, producing corrupted bytes. Now:
- s3ObjectState gains uploadID; HandleObjectManifest sets it.
- New filterChunksForManifest takes the chunkPaths map and the
manifest's uploadID, returns only matching chunks sorted by
(partNo, partVersion, chunkNo). Stale-uploadID chunks never
enter the assembled body.
Test: TestS3_StaleUploadIDChunksFilteredFromAssembledBody.
CRITICAL: incomplete-uploads file truncated per record (Codex P2
#318, Gemini HIGH+MEDIUM #318).
HandleIncompleteUpload re-opened records.jsonl on every call;
openJSONL uses os.Create which truncates. Only the last record
survived per bucket. Now:
- s3BucketState carries an incompleteUploadsJL *jsonlFile lazily
opened on the first record and cached.
- flushBucket closes it and surfaces the error (was silently
ignored).
Test: TestS3_IncompleteUploadsAppendsAcrossCalls (3 records, 3
lines on disk).
QUALITY:
- Gemini MEDIUM #285 (MkdirAll per blob): s3ObjectState gains a
scratchDirCreated bool; HandleBlob runs MkdirAll once.
- Gemini MEDIUM #318 (closeJSONL error ignored): keymap and
incomplete-uploads writers now surface close errors via
closeBucketKeymap / explicit closeJSONL return-check.
- Gemini MEDIUM #386 (includeOrphans flag ignored): orphan chunks
for objects without manifests now write to
<bucket>/_orphans/<encoded-object>/<chunk>.bin under
WithIncludeOrphans(true). Test:
TestS3_OrphanChunksWrittenWhenIncludeOrphans.
Also removed the now-unused attachManifestParts placeholder and
sortChunkKeys helper; their logic moved into
filterChunksForManifest which combines uploadID filtering with the
sort.
@bootjpbootjp mentioned this pull request Apr 30, 2026
Merged
3 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@bootjpbootjpbootjp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@bootjpbootjp

Copilot code reviewCopilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add metrics output for prometheus

2 participants

@bootjp