Skip to content

perf: experiment with striped classic histogram accumulator - #2329

Draft
zeitlinger wants to merge 2 commits into
prometheus:mainfrom
zeitlinger:agent/classic-only-striped-accumulator
Draft

perf: experiment with striped classic histogram accumulator#2329
zeitlinger wants to merge 2 commits into
prometheus:mainfrom
zeitlinger:agent/classic-only-striped-accumulator

Conversation

@zeitlinger

@zeitlingerzeitlinger commented Jul 23, 2026

Copy link
Copy Markdown
Member

Summary

This is an experimental alternate implementation for benchmarking against main and the synchronized candidate in #2252. It is not being presented as production-ready yet.

For classic-only histograms, this replaces the existing Buffer/LongAdder recording path with an epoch-striped, double-buffered per-thread accumulator. Each recording thread owns a cell, observations update plain thread-confined bucket/count/sum fields, and collection advances a global epoch before draining the inactive buffers. This avoids a global synchronized (DataPoint) monitor on the shared four-thread observe path while preserving coherent bucket/count/sum snapshots. getCount() and getSum() use the same snapshot mechanism.

Native-only and hybrid histograms keep their existing implementation. The public API is unchanged.

Local evidence

  • mise run lint
  • ./mvnw -pl prometheus-metrics-core -Dtest=HistogramTest test — 35 tests passed
  • focused concurrent collect/lifecycle tests repeated 5 times
  • ./mvnw -pl prometheus-metrics-core -am verify — reactor passed; core ran 157 tests
  • git diff --check
  • representative local JMH smoke run (JDK 25.0.3, 4 threads, 1 fork, 3x1s warmup, 5x1s measurement):
    • HistogramBenchmark.prometheusClassic: 40,944.515 ± 302.666 ops/s

The local JMH number is a smoke result, not a controlled cross-branch conclusion. The benchmark workflow is being requested so this branch and main are measured on the same runner.

Added concurrency coverage

  • collection while eight shared observers record concurrently
  • monotonic and coherent bucket/count/sum snapshots
  • exact final getCount() / getSum() values
  • retention of observations from short-lived recording threads
  • clear() lifecycle behavior for old and newly-created data points

Remaining hardening before merge consideration

  • Review/validate the epoch handoff under the Java Memory Model, ideally with dedicated jcstress coverage rather than relying only on repeated JUnit stress tests.
  • Decide how to bound or reclaim retained per-thread cells for workloads with high thread churn; cells currently live for the data point lifetime so completed-thread observations cannot be lost.
  • Measure scrape latency and the spin-wait behavior when a recording thread is delayed after entering an epoch.
  • Measure the cost of getCount() / getSum(), which currently perform an epoch flip and cell scan to return exact accumulated values.
  • Run the full CI benchmark matrix and test additional supported JDKs/architectures before treating this as mergeable.

Related: #2252

@zeitlinger
perf: experiment with striped classic histogram accumulator
0e08d51
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlingerzeitlinger mentioned this pull request Jul 23, 2026
Open
@zeitlinger
docs: update core API diff
6fe2995
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlingerzeitlinger reopened this Jul 23, 2026
@github-actions

github-actionsBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

  • prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

1 similar comment
@github-actions

github-actionsBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

  • prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

@github-actions

github-actionsBot commented Jul 23, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark results

Benchmark run succeeded for 6fe299560f57865c3ca33d399163a0639f84b47e.

Prometheus Java Client Benchmarks

Run Information

  • Date: 2026-07-23T10:46:17Z
  • Commit:6fe2995
  • JDK: 25.0.3 (OpenJDK 64-Bit Server VM)
  • Benchmark config: 3 fork(s), 3 warmup, 5 measurement, 4 threads
  • Hardware: AMD EPYC 9V74 80-Core Processor, 4 cores, 16 GB RAM
  • OS: Linux 6.17.0-1020-azure

Comparison with base

  • Head:6fe2995
  • Base:1ca2716
  • Change: positive means the PR is faster than base.
  • Note: Base and head are compared on the same runner within each topic.
BenchmarkPRBaseChangeResult
CounterBenchmark.codahaleIncNoLabels44.11K44.37K-0.6%within noise
CounterBenchmark.openTelemetryAdd3.25K4.53K-28.2%slower
CounterBenchmark.openTelemetryInc4.42K4.49K-1.6%within noise
CounterBenchmark.openTelemetryIncNoLabels3.59K4.70K-23.7%within noise
CounterBenchmark.prometheusAdd48.73K48.49K+0.5%within noise
CounterBenchmark.prometheusInc60.03K59.25K+1.3%within noise
CounterBenchmark.prometheusNoLabelsInc51.95K51.05K+1.8%faster
CounterBenchmark.simpleclientAdd6.12K5.92K+3.4%faster
CounterBenchmark.simpleclientInc6.16K6.20K-0.5%within noise
CounterBenchmark.simpleclientNoLabelsInc5.92K5.92K+0.1%within noise
HistogramBenchmark.openTelemetryClassic766.00766.29-0.0%within noise
HistogramBenchmark.openTelemetryExponential628.37704.07-10.8%within noise
HistogramBenchmark.prometheusClassic11.59K7.18K+61.4%faster
HistogramBenchmark.prometheusClassicPerThread12.23K12.58K-2.8%slower
HistogramBenchmark.prometheusClassicSingleThread4.40K4.55K-3.2%slower
HistogramBenchmark.prometheusNative2.81K2.82K-0.3%within noise
HistogramBenchmark.simpleclient4.41K4.36K+1.2%within noise
HistogramTextFormatBenchmark.openMetricsWriteToNull27.07K27.24K-0.6%within noise
HistogramTextFormatBenchmark.prometheusWriteToNull27.56K27.63K-0.2%within noise
TextFormatUtilBenchmark.openMetricsWriteToByteArray526.71K533.80K-1.3%within noise
TextFormatUtilBenchmark.openMetricsWriteToNull541.08K546.14K-0.9%within noise
TextFormatUtilBenchmark.prometheusWriteToByteArray568.30K573.63K-0.9%within noise
TextFormatUtilBenchmark.prometheusWriteToNull582.87K570.66K+2.1%within noise

Results for PR head

CounterBenchmark

BenchmarkScoreErrorUnitsWithin run
prometheusInc60.03K± 926.53ops/sfastest
prometheusNoLabelsInc51.95K± 18.60ops/s1.2x slower
prometheusAdd48.73K± 396.08ops/s1.2x slower
codahaleIncNoLabels44.11K± 115.85ops/s1.4x slower
simpleclientInc6.16K± 130.99ops/s9.7x slower
simpleclientAdd6.12K± 22.12ops/s9.8x slower
simpleclientNoLabelsInc5.92K± 18.44ops/s10x slower
openTelemetryInc4.42K± 1.19Kops/s14x slower
openTelemetryIncNoLabels3.59K± 261.88ops/s17x slower
openTelemetryAdd3.25K± 151.58ops/s18x slower

HistogramBenchmark

BenchmarkScoreErrorUnitsWithin run
prometheusClassicPerThread12.23K± 292.11ops/sfastest
prometheusClassic11.59K± 428.04ops/s1.1x slower
simpleclient4.41K± 76.08ops/s2.8x slower
prometheusClassicSingleThread4.40K± 19.23ops/s2.8x slower
prometheusNative2.81K± 252.77ops/s4.4x slower
openTelemetryClassic766.00± 23.21ops/s16x slower
openTelemetryExponential628.37± 82.07ops/s19x slower

HistogramTextFormatBenchmark

BenchmarkScoreErrorUnitsWithin run
prometheusWriteToNull27.56K± 317.22ops/sfastest
openMetricsWriteToNull27.07K± 332.83ops/s1.0x slower

TextFormatUtilBenchmark

BenchmarkScoreErrorUnitsWithin run
prometheusWriteToNull582.87K± 12.95Kops/sfastest
prometheusWriteToByteArray568.30K± 5.31Kops/s1.0x slower
openMetricsWriteToNull541.08K± 12.50Kops/s1.1x slower
openMetricsWriteToByteArray526.71K± 15.29Kops/s1.1x slower

Raw Results

Benchmark Mode Cnt Score Error Units
CounterBenchmark.codahaleIncNoLabels thrpt 15 44112.334 ± 115.847 ops/s
CounterBenchmark.openTelemetryAdd thrpt 15 3252.670 ± 151.579 ops/s
CounterBenchmark.openTelemetryInc thrpt 15 4416.392 ± 1185.330 ops/s
CounterBenchmark.openTelemetryIncNoLabels thrpt 15 3586.093 ± 261.881 ops/s
CounterBenchmark.prometheusAdd thrpt 15 48729.646 ± 396.084 ops/s
CounterBenchmark.prometheusInc thrpt 15 60027.803 ± 926.533 ops/s
CounterBenchmark.prometheusNoLabelsInc thrpt 15 51952.103 ± 18.603 ops/s
CounterBenchmark.simpleclientAdd thrpt 15 6118.802 ± 22.118 ops/s
CounterBenchmark.simpleclientInc thrpt 15 6163.636 ± 130.990 ops/s
CounterBenchmark.simpleclientNoLabelsInc thrpt 15 5922.243 ± 18.435 ops/s
HistogramBenchmark.openTelemetryClassic thrpt 15 765.996 ± 23.210 ops/s
HistogramBenchmark.openTelemetryExponential thrpt 15 628.374 ± 82.069 ops/s
HistogramBenchmark.prometheusClassic thrpt 15 11589.781 ± 428.043 ops/s
HistogramBenchmark.prometheusClassicPerThread thrpt 15 12228.590 ± 292.114 ops/s
HistogramBenchmark.prometheusClassicSingleThread thrpt 15 4398.687 ± 19.226 ops/s
HistogramBenchmark.prometheusNative thrpt 15 2808.686 ± 252.769 ops/s
HistogramBenchmark.simpleclient thrpt 15 4410.533 ± 76.084 ops/s
HistogramTextFormatBenchmark.openMetricsWriteToNull thrpt 15 27071.116 ± 332.829 ops/s
HistogramTextFormatBenchmark.prometheusWriteToNull thrpt 15 27561.877 ± 317.215 ops/s
TextFormatUtilBenchmark.openMetricsWriteToByteArray thrpt 15 526713.833 ± 15292.911 ops/s
TextFormatUtilBenchmark.openMetricsWriteToNull thrpt 15 541078.028 ± 12499.428 ops/s
TextFormatUtilBenchmark.prometheusWriteToByteArray thrpt 15 568297.769 ± 5312.819 ops/s
TextFormatUtilBenchmark.prometheusWriteToNull thrpt 15 582874.507 ± 12954.847 ops/s

Notes

  • Score = Throughput in operations per second (higher is better)
  • Error = 99.9% confidence interval
  • Comparison with base uses JMH confidence intervals when available; overlapping intervals are marked "within noise".
  • Within run compares benchmarks in the same result set, not against the base commit.

Benchmark Descriptions

BenchmarkDescription
CounterBenchmarkCounter increment performance: Prometheus, OpenTelemetry, simpleclient, Codahale
HistogramBenchmarkHistogram observation performance (classic vs native/exponential)
TextFormatUtilBenchmarkMetric exposition format writing speed

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fstabfstabAwaiting requested review from fstabfstab will be requested when the pull request is marked ready for reviewfstab is a code owner
@dhoarddhoardAwaiting requested review from dhoarddhoard will be requested when the pull request is marked ready for reviewdhoard is a code owner
@jaydelucajaydelucaAwaiting requested review from jaydelucajaydeluca will be requested when the pull request is marked ready for reviewjaydeluca is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

api-changebenchmark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zeitlinger