From 671f1d47a3c3f5878de0120c8bc2d5b276acbece Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 03:22:29 +0000 Subject: [PATCH 1/4] Chart performance per release, and show it in the README [minor] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README now opens on a picture of what each release did to allocation and speed, drawn from numbers this repository keeps rather than from a run someone remembered to do. Three pieces. `BaselineBenchmarks` measures a fixed integer workload that touches none of this library. `scripts/benchmark_history.py` reads BenchmarkDotNet's JSON into a committed history keyed by version and draws it as SVG. `benchmark-history.yml` measures a published release and adds a point, or backfills a list of refs on demand. The awkward part is that a benchmark chart across releases is mostly a chart of CI runners. Each release is measured in its own job, and the difference between the hosts a job can land on is larger than most releases are. Two things address it rather than hide it. Allocation is charted because it is exact: the same code allocates the same bytes anywhere, so a step in the top row is always real. Times are divided by the reference workload measured in the same job, which cancels most of the difference between machines, and the README says plainly that what remains is indicative. A backfill measures every ref in one job for the same reason — points gathered on one host are comparable as they stand. The chart is seeded with 1.9.0 and 2.0.0 through 2.0.4, measured here rather than on a runner, so the README works the moment this merges. Each entry records the CPU it was measured on, and ingesting a version replaces its entry, so re-running the backfill in CI overwrites these with runner-measured points. It reproduces the value type result independently: every allocation drop at 2.0.0 is exactly 40 bytes, the object header, and ToDouble falls from 0.045 to 0.010 of the reference — the same 79% the one-off comparison on #72 measured. The README's Performance section moved up to sit after Features, rather than a second Performance heading being added, which would have taken the anchor. Four things the tests caught, all of which would have failed quietly: - `git diff --quiet` does not see an untracked file, so the first run would have committed nothing and reported success. - `git show -s --format=%cs` on an annotated tag prints the whole tag object, so every tagged entry recorded the tagger message as its date. - `git rev-parse --short v2.0.0` resolves to the tag object, not the commit. - A release event checks out the tag, so committing results from that checkout asks the default branch to move backwards. Both paths now check out the default branch and measure their ref through a worktree. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017jrnV7N94UGL8fDRRE8Xt8 --- .github/workflows/benchmark-history.yml | 209 ++++++ .gitignore | 4 + .../BaselineBenchmarks.cs | 59 ++ README.md | 59 +- docs/benchmarks/history.json | 611 ++++++++++++++++++ docs/benchmarks/performance-dark.svg | 177 +++++ docs/benchmarks/performance.svg | 177 +++++ scripts/benchmark_history.py | 462 +++++++++++++ 8 files changed, 1736 insertions(+), 22 deletions(-) create mode 100644 .github/workflows/benchmark-history.yml create mode 100644 PreciseNumber.Benchmarks/BaselineBenchmarks.cs create mode 100644 docs/benchmarks/history.json create mode 100644 docs/benchmarks/performance-dark.svg create mode 100644 docs/benchmarks/performance.svg create mode 100644 scripts/benchmark_history.py diff --git a/.github/workflows/benchmark-history.yml b/.github/workflows/benchmark-history.yml new file mode 100644 index 0000000..29e508a --- /dev/null +++ b/.github/workflows/benchmark-history.yml @@ -0,0 +1,209 @@ +name: Benchmark History + +# Measures a small, fixed set of benchmarks once per release, appends the numbers to a committed +# history file, and redraws the chart the README shows. Deliberately separate from benchmarks.yml, +# which exists to get a full ad-hoc run on demand and publishes nothing. +# +# Two ways in: +# * a published release, which measures that version and adds one point; +# * a manual dispatch listing refs, which measures each of them in ONE job and backfills. +# +# The backfill running as a single job is the point rather than an optimisation. Separate runs land +# on different CI hosts, and the difference between an x86-64-v3 and a v4 runner is larger than +# most releases are, so points gathered in separate jobs are not comparable as raw times. Within +# one job they are. Across jobs, BaselineBenchmarks is what ties them together -- see its remarks. + +on: + release: + types: [published] + workflow_dispatch: + inputs: + refs: + description: "Space-separated refs to backfill, oldest first (tags, branches, or SHAs)" + required: false + default: "cd9a8227793bbd6ea791be7f6c0579ae1242eec5 v2.0.0 v2.0.1 v2.0.2 v2.0.3 v2.0.4" + type: string + labels: + description: "Optional space-separated version labels matching refs, when a ref is not a version" + required: false + default: "1.9.0 2.0.0 2.0.1 2.0.2 2.0.3 2.0.4" + type: string + +permissions: + contents: write + +concurrency: + group: benchmark-history + cancel-in-progress: false + +env: + DOTNET_VERSION: "10.0" + HISTORY: docs/benchmarks/history.json + CHART: docs/benchmarks/performance.svg + # The set drawn in the README. Name globs rather than a [BenchmarkCategory], because the backfill + # runs this same filter against older checkouts that predate any attribute added today. + HEADLINE_FILTER: >- + *ArithmeticBenchmarks.Add + *ArithmeticBenchmarks.Multiply + *ArithmeticBenchmarks.Divide + *ComparisonBenchmarks.CompareTo + *ConstructionBenchmarks.Sanitizing + *TextBenchmarks.Parse + *ConversionBenchmarks.ToDouble + # Short runs: three iterations is enough for a trend line, and a release should not tie up a + # runner for half an hour. benchmarks.yml is still there for a full-length run. + BENCHMARK_JOB: short + +jobs: + measure: + name: Measure and chart + runs-on: ubuntu-latest + timeout-minutes: 180 + + steps: + - name: Checkout Repository + uses: actions/checkout@v7 + with: + # The default branch, not the released tag. The results are committed back here, and a + # release event would otherwise leave the checkout detached at the tag, so the push at + # the end would be asking the default branch to move backwards. The tag itself is + # measured through a worktree below, exactly as a backfill ref is. + ref: ${{ github.event.repository.default_branch }} + fetch-depth: 0 + + - name: Setup .NET SDK ${{ env.DOTNET_VERSION }} + uses: actions/setup-dotnet@v6 + with: + dotnet-version: ${{ env.DOTNET_VERSION }}.x + + # Measured from this checkout, and stamped onto every entry this job produces. Older refs do + # not carry BaselineBenchmarks, and do not need to: everything measured in this job shares + # one runner, so one reading of that runner describes all of them. + - name: Measure the reference workload + id: baseline + shell: bash + run: | + set -euo pipefail + dotnet run -c Release --project PreciseNumber.Benchmarks -- \ + --filter '*BaselineBenchmarks.ReferenceWork' \ + --job "$BENCHMARK_JOB" \ + --artifacts "$GITHUB_WORKSPACE/bench/baseline" + ns=$(python3 - <<'PY' + import glob, json + path = glob.glob("bench/baseline/results/*BaselineBenchmarks-report-full.json")[0] + with open(path, encoding="utf-8-sig") as handle: + document = json.load(handle) + print(round(document["Benchmarks"][0]["Statistics"]["Mean"], 4)) + PY + ) + echo "Reference workload: $ns ns" + echo "ns=$ns" >> "$GITHUB_OUTPUT" + + - name: Measure the released version + if: github.event_name == 'release' + shell: bash + env: + VERSION: ${{ github.event.release.tag_name }} + run: | + set -euo pipefail + version="${VERSION#v}" + work="$GITHUB_WORKSPACE/backfill/$version" + git worktree add --detach "$work" "$VERSION" + + (cd "$work" && dotnet run -c Release --project PreciseNumber.Benchmarks -- \ + --filter $HEADLINE_FILTER \ + --job "$BENCHMARK_JOB" \ + --artifacts "$GITHUB_WORKSPACE/bench/$version") + + python3 scripts/benchmark_history.py ingest \ + --history "$HISTORY" \ + --results "bench/$version" \ + --version "$version" \ + --commit "$(git rev-parse --short "$VERSION^{commit}")" \ + --date "$(git log -1 --format=%cs "$VERSION")" \ + --run-id "${{ github.run_id }}" \ + --baseline-ns "${{ steps.baseline.outputs.ns }}" + + git worktree remove --force "$work" + + - name: Measure each backfill ref + if: github.event_name == 'workflow_dispatch' + shell: bash + env: + REFS: ${{ inputs.refs }} + LABELS: ${{ inputs.labels }} + BASELINE_NS: ${{ steps.baseline.outputs.ns }} + run: | + set -euo pipefail + read -ra refs <<< "$REFS" + read -ra labels <<< "$LABELS" + + for index in "${!refs[@]}"; do + ref="${refs[$index]}" + label="${labels[$index]:-${ref#v}}" + work="$GITHUB_WORKSPACE/backfill/$label" + + echo "::group::$label ($ref)" + rm -rf "$work" + git worktree add --detach "$work" "$ref" + + if [ ! -f "$work/PreciseNumber.Benchmarks/PreciseNumber.Benchmarks.csproj" ]; then + echo "::warning::$ref has no benchmark project; skipping" + git worktree remove --force "$work" + echo "::endgroup::" + continue + fi + + # Each ref is measured by its own benchmark sources. Between 2.0.0 and now those + # sources are unchanged, so this compares library versions rather than harnesses. + (cd "$work" && dotnet run -c Release --project PreciseNumber.Benchmarks -- \ + --filter $HEADLINE_FILTER \ + --job "$BENCHMARK_JOB" \ + --artifacts "$GITHUB_WORKSPACE/bench/$label") + + python3 scripts/benchmark_history.py ingest \ + --history "$HISTORY" \ + --results "bench/$label" \ + --version "$label" \ + --commit "$(git rev-parse --short "$ref^{commit}")" \ + --date "$(git log -1 --format=%cs "$ref")" \ + --run-id "${{ github.run_id }}" \ + --baseline-ns "$BASELINE_NS" + + git worktree remove --force "$work" + echo "::endgroup::" + done + + - name: Redraw the chart + shell: bash + run: python3 scripts/benchmark_history.py render --history "$HISTORY" --out "$CHART" + + - name: Commit the history and the chart + shell: bash + run: | + set -euo pipefail + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + # Staged first, then compared against the index: on the first run these files are new, + # and `git diff` alone does not see an untracked file, so the run would push nothing and + # still report success. + git add "$HISTORY" "${CHART%.svg}"*.svg + if git diff --cached --quiet; then + echo "Nothing changed." + exit 0 + fi + # [skip ci] so that committing results does not start the pipeline over again. + git commit -m "[bot][skip ci] Update benchmark history" + branch="${{ github.event.repository.default_branch }}" + # Another release may have landed while this job was measuring. + git pull --rebase origin "$branch" + git push origin "HEAD:$branch" + + - name: Upload the raw reports + if: always() + uses: actions/upload-artifact@v7 + with: + name: benchmark-history-${{ github.run_id }} + path: bench/ + retention-days: 30 + if-no-files-found: warn diff --git a/.gitignore b/.gitignore index dc0470a..e7ff079 100644 --- a/.gitignore +++ b/.gitignore @@ -58,6 +58,10 @@ dlldata.c # Benchmark Results BenchmarkDotNet.Artifacts/ +# Scratch directories the Benchmark History workflow writes its runs into. +bench/ +backfill/ + # .NET Core project.lock.json project.fragment.lock.json diff --git a/PreciseNumber.Benchmarks/BaselineBenchmarks.cs b/PreciseNumber.Benchmarks/BaselineBenchmarks.cs new file mode 100644 index 0000000..897d05a --- /dev/null +++ b/PreciseNumber.Benchmarks/BaselineBenchmarks.cs @@ -0,0 +1,59 @@ +// Copyright (c) 2023-2026 ktsu-dev contributors + +namespace ktsu.PreciseNumber.Benchmarks; + +using BenchmarkDotNet.Attributes; + +/// +/// Measures a fixed workload that touches none of this library, so that timings taken on +/// different machines can be compared. +/// +/// +/// Every release is benchmarked in its own CI job, and a job lands on whichever shared runner is +/// free — an x86-64-v3 or v4 host, at whatever clock its neighbours leave it. That difference is +/// routinely larger than the changes a release makes, so a chart of raw times across releases +/// mostly plots the runner. +/// +/// This benchmark is the fixed point that makes the rest comparable. It is integer arithmetic over +/// a value the JIT cannot fold away, chosen because it has no allocation, no library code, and no +/// dependence on anything that changes between versions — so its measured time is a reading of the +/// machine and nothing else. Dividing a benchmark's time by this one's, taken in the same job, +/// cancels most of the difference between hosts. `scripts/benchmark_history.py` records it on +/// every entry and plots the ratio rather than the nanoseconds. +/// +/// +/// It follows that this method's body must never change. Editing it silently rescales every +/// comparison drawn against history recorded before the edit. +/// +/// +[MemoryDiagnoser] +public class BaselineBenchmarks +{ + // Read from a field rather than written as a literal, so that the loop cannot be constant + // folded into its own answer at JIT time. + private ulong seed; + + /// + /// Sets the starting value. + /// + [GlobalSetup] + public void Setup() => seed = 0xcbf29ce484222325; + + /// + /// Mixes a counter with a multiply-xor-shift step, the way a non-cryptographic hash does. + /// + /// The accumulated value, returned so that nothing here is dead code. + [Benchmark] + public ulong ReferenceWork() + { + ulong accumulator = seed; + + for (int i = 0; i < 256; i++) + { + accumulator = (accumulator ^ (ulong)i) * 0x100000001b3; + accumulator ^= accumulator >> 29; + } + + return accumulator; + } +} diff --git a/README.md b/README.md index 7ba9185..fe4b64f 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ A high-precision numeric type for .NET that provides arbitrary precision arithme - [Features](#features) +- [Performance](#performance) + - [Getting Started](#getting-started) - [Installation](#installation) @@ -56,8 +58,6 @@ A high-precision numeric type for .NET that provides arbitrary precision arithme - [Limitations](#limitations) -- [Performance](#performance) - - [API Reference](#api-reference) - [PreciseNumber Class](#precisenumber-class) @@ -86,6 +86,41 @@ A high-precision numeric type for .NET that provides arbitrary precision arithme - **Balanced Performance**: The design prioritizes accuracy and precision while maintaining reasonable performance. For calculations where extreme precision matters more than raw speed, PreciseNumber delivers excellent results, though built-in numeric types remain faster for standard precision needs. +## Performance + +Values are immutable value types. Every operation returns a new value, but that value lives inline +in its variable, field, or array element, so the only heap allocation is the `BigInteger` digit +array, and a significand that fits in an `int` doesn't need one. Cost therefore tracks the number +of significant digits rather than the magnitude of the value, and allocation matters as much as +raw speed. + + + + Allocated bytes per operation, and time relative to a fixed reference workload, for each PreciseNumber release + + +Every release measures a fixed set of benchmarks and adds a point to the chart above; the numbers +behind it are in [`docs/benchmarks/history.json`](docs/benchmarks/history.json). + +Read the two halves differently. **Allocation is exact** — the same code allocates the same bytes on +any machine, so a step in the top row is always a real change. **Time is measured on shared CI +runners**, where the host a job happens to land on varies more than most releases do, so each time +is divided by a reference workload measured in the same job. That cancels most of the difference +between machines; what is left is indicative rather than precise, and a small wobble between two +releases is more likely the runner than the library. + +The repository carries a [BenchmarkDotNet suite](PreciseNumber.Benchmarks/README.md) covering +construction, comparison, arithmetic, rounding, text conversion and primitive conversion, each +parameterised across 8, 30 and 200 significant digits: + +```bash +dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*ArithmeticBenchmarks*' +``` + +Run it before and after any change to the library's internals. A full run can also be started +from the **Benchmarks** workflow in GitHub Actions, which archives the reports against the commit +that produced them. + # Getting Started ## Installation @@ -394,26 +429,6 @@ three-argument overload when you want something other than that. - Converting from `double`, `float`, or `Half` keeps the shortest digits that round-trip, so converting back gives the original value, and `0.3048` stays exactly 0.3048. The result of `0.1 + 0.2` in `double` arrives as 0.30000000000000004, because that's the value the `double` holds -## Performance - -Values are immutable value types. Every operation returns a new value, but that value lives inline -in its variable, field, or array element, so the only heap allocation is the `BigInteger` digit -array, and a significand that fits in an `int` doesn't need one. Cost therefore tracks the number -of significant digits rather than the magnitude of the value, and allocation matters as much as -raw speed. - -The repository carries a [BenchmarkDotNet suite](PreciseNumber.Benchmarks/README.md) covering -construction, comparison, arithmetic, rounding, text conversion and primitive conversion, each -parameterised across 8, 30 and 200 significant digits: - -```bash -dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*ArithmeticBenchmarks*' -``` - -Run it before and after any change to the library's internals. A full run can also be started -from the **Benchmarks** workflow in GitHub Actions, which archives the reports against the commit -that produced them. - ## API Reference ### PreciseNumber Class diff --git a/docs/benchmarks/history.json b/docs/benchmarks/history.json new file mode 100644 index 0000000..a7f8fc2 --- /dev/null +++ b/docs/benchmarks/history.json @@ -0,0 +1,611 @@ +{ + "schemaVersion": 1, + "entries": [ + { + "version": "1.9.0", + "commit": "cd9a822", + "date": "2026-09-13", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 456.3006, + "runId": "local-backfill", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 118.4378, + "allocatedBytes": 104 + }, + "Digits=30": { + "meanNs": 157.0446, + "allocatedBytes": 120 + }, + "Digits=200": { + "meanNs": 401.5004, + "allocatedBytes": 264 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 408.9082, + "allocatedBytes": 192 + }, + "Digits=30": { + "meanNs": 597.664, + "allocatedBytes": 240 + }, + "Digits=200": { + "meanNs": 3397.0807, + "allocatedBytes": 568 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 85.9557, + "allocatedBytes": 72 + }, + "Digits=30": { + "meanNs": 180.616, + "allocatedBytes": 96 + }, + "Digits=200": { + "meanNs": 1017.6489, + "allocatedBytes": 232 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 4.9821, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 4.7751, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 5.1426, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 33.1727, + "allocatedBytes": 40 + }, + "Digits=30": { + "meanNs": 76.8865, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 257.1197, + "allocatedBytes": 40 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 20.6804, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 205.5837, + "allocatedBytes": 40 + }, + "Digits=30": { + "meanNs": 489.1961, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 2436.9512, + "allocatedBytes": 152 + } + } + } + }, + { + "version": "2.0.0", + "commit": "da3aac6", + "date": "2026-09-13", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 456.3006, + "runId": "local-backfill", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 102.2084, + "allocatedBytes": 64 + }, + "Digits=30": { + "meanNs": 144.8577, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 401.2804, + "allocatedBytes": 224 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 425.5046, + "allocatedBytes": 152 + }, + "Digits=30": { + "meanNs": 636.3521, + "allocatedBytes": 200 + }, + "Digits=200": { + "meanNs": 3408.2489, + "allocatedBytes": 528 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 66.8426, + "allocatedBytes": 32 + }, + "Digits=30": { + "meanNs": 167.6952, + "allocatedBytes": 56 + }, + "Digits=200": { + "meanNs": 1011.6339, + "allocatedBytes": 192 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 3.7839, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 3.7399, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 3.5946, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 19.1305, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 64.8693, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 245.6405, + "allocatedBytes": 0 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 5.6483, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 185.9653, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 478.1012, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 2545.4556, + "allocatedBytes": 112 + } + } + } + }, + { + "version": "2.0.1", + "commit": "b2427f0", + "date": "2026-09-14", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 456.3006, + "runId": "local-backfill", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 98.0732, + "allocatedBytes": 64 + }, + "Digits=30": { + "meanNs": 143.3102, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 388.1712, + "allocatedBytes": 224 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 421.3311, + "allocatedBytes": 152 + }, + "Digits=30": { + "meanNs": 639.3545, + "allocatedBytes": 200 + }, + "Digits=200": { + "meanNs": 2377.5382, + "allocatedBytes": 528 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 66.1012, + "allocatedBytes": 32 + }, + "Digits=30": { + "meanNs": 166.4438, + "allocatedBytes": 56 + }, + "Digits=200": { + "meanNs": 1005.4831, + "allocatedBytes": 192 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 3.6457, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 3.9055, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 3.8933, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 19.1771, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 63.2906, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 241.3681, + "allocatedBytes": 0 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 5.7523, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 187.7729, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 485.3833, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 2510.739, + "allocatedBytes": 112 + } + } + } + }, + { + "version": "2.0.2", + "commit": "8bf324f", + "date": "2026-09-14", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 456.3006, + "runId": "local-backfill", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 106.141, + "allocatedBytes": 64 + }, + "Digits=30": { + "meanNs": 151.7249, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 412.2945, + "allocatedBytes": 224 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 428.2106, + "allocatedBytes": 152 + }, + "Digits=30": { + "meanNs": 647.3638, + "allocatedBytes": 200 + }, + "Digits=200": { + "meanNs": 2435.1219, + "allocatedBytes": 528 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 72.0185, + "allocatedBytes": 32 + }, + "Digits=30": { + "meanNs": 172.6775, + "allocatedBytes": 56 + }, + "Digits=200": { + "meanNs": 1021.6319, + "allocatedBytes": 192 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 4.1897, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 3.6436, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 4.4223, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 19.5526, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 64.9609, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 243.2625, + "allocatedBytes": 0 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 4.7889, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 188.7315, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 502.7654, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 2469.2835, + "allocatedBytes": 112 + } + } + } + }, + { + "version": "2.0.3", + "commit": "6722ff9", + "date": "2026-09-14", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 456.3006, + "runId": "local-backfill", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 99.4048, + "allocatedBytes": 64 + }, + "Digits=30": { + "meanNs": 149.9368, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 417.451, + "allocatedBytes": 224 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 436.516, + "allocatedBytes": 152 + }, + "Digits=30": { + "meanNs": 641.4446, + "allocatedBytes": 200 + }, + "Digits=200": { + "meanNs": 3476.5374, + "allocatedBytes": 528 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 72.2699, + "allocatedBytes": 32 + }, + "Digits=30": { + "meanNs": 171.918, + "allocatedBytes": 56 + }, + "Digits=200": { + "meanNs": 1027.3061, + "allocatedBytes": 192 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 4.2138, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 3.828, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 3.5863, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 20.0353, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 62.9153, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 241.841, + "allocatedBytes": 0 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 5.3217, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 192.7158, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 472.4866, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 2528.1408, + "allocatedBytes": 112 + } + } + } + }, + { + "version": "2.0.4", + "commit": "c265bdd", + "date": "2026-09-15", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 456.3006, + "runId": "local-backfill", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 102.0636, + "allocatedBytes": 64 + }, + "Digits=30": { + "meanNs": 149.3164, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 409.0072, + "allocatedBytes": 224 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 425.7428, + "allocatedBytes": 152 + }, + "Digits=30": { + "meanNs": 642.9308, + "allocatedBytes": 200 + }, + "Digits=200": { + "meanNs": 3448.2813, + "allocatedBytes": 528 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 69.2106, + "allocatedBytes": 32 + }, + "Digits=30": { + "meanNs": 173.0146, + "allocatedBytes": 56 + }, + "Digits=200": { + "meanNs": 1028.6434, + "allocatedBytes": 192 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 4.1475, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 3.9174, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 4.1474, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 19.5191, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 63.7933, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 243.2835, + "allocatedBytes": 0 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 4.7245, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 185.7342, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 484.1852, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 2339.7804, + "allocatedBytes": 112 + } + } + } + } + ] +} diff --git a/docs/benchmarks/performance-dark.svg b/docs/benchmarks/performance-dark.svg new file mode 100644 index 0000000..1f683f6 --- /dev/null +++ b/docs/benchmarks/performance-dark.svg @@ -0,0 +1,177 @@ + + + +PreciseNumber performance by release +6 releases · newest 2.0.4 · 2026-09-15 + +Allocated bytes per operation +Deterministic: the same code allocates the same bytes on any machine. +Add (30 digits) + + + + + + + + +80 B +120 B +Multiply (30 digits) + + + + + + + + +56 B +96 B +Divide (30 digits) + + + + + + + + +200 B +240 B +CompareTo (30 digits) + + + + + + + + +0 B +0 B +Construct (30 digits) + + + + + + + + +0 B +40 B +Parse (30 digits) + + + + + + + + +40 B +80 B +ToDouble + + + + + + + + +0 B +0 B + +Time, as a multiple of a fixed reference workload +Divided by a reference loop measured in the same job, which cancels most of the difference between CI runners. Lower is faster. +Add (30 digits) + + + + + + + + +0.327× +0.344× +Multiply (30 digits) + + + + + + + + +0.379× +0.396× +Divide (30 digits) + + + + + + + + +1.41× +1.31× +CompareTo (30 digits) + + + + + + + + +0.0086× +0.0105× +Construct (30 digits) + + + + + + + + +0.140× +0.168× +Parse (30 digits) + + + + + + + + +1.06× +1.07× +ToDouble + + + + + + + + +0.0104× +0.0453× +releases, oldest to newest: 1.9.0 → 2.0.0 → 2.0.1 → 2.0.2 → 2.0.3 → 2.0.4 +Measured on Intel Xeon Processor 2.80GHz. Full tables: PreciseNumber.Benchmarks. + diff --git a/docs/benchmarks/performance.svg b/docs/benchmarks/performance.svg new file mode 100644 index 0000000..8e6fac5 --- /dev/null +++ b/docs/benchmarks/performance.svg @@ -0,0 +1,177 @@ + + + +PreciseNumber performance by release +6 releases · newest 2.0.4 · 2026-09-15 + +Allocated bytes per operation +Deterministic: the same code allocates the same bytes on any machine. +Add (30 digits) + + + + + + + + +80 B +120 B +Multiply (30 digits) + + + + + + + + +56 B +96 B +Divide (30 digits) + + + + + + + + +200 B +240 B +CompareTo (30 digits) + + + + + + + + +0 B +0 B +Construct (30 digits) + + + + + + + + +0 B +40 B +Parse (30 digits) + + + + + + + + +40 B +80 B +ToDouble + + + + + + + + +0 B +0 B + +Time, as a multiple of a fixed reference workload +Divided by a reference loop measured in the same job, which cancels most of the difference between CI runners. Lower is faster. +Add (30 digits) + + + + + + + + +0.327× +0.344× +Multiply (30 digits) + + + + + + + + +0.379× +0.396× +Divide (30 digits) + + + + + + + + +1.41× +1.31× +CompareTo (30 digits) + + + + + + + + +0.0086× +0.0105× +Construct (30 digits) + + + + + + + + +0.140× +0.168× +Parse (30 digits) + + + + + + + + +1.06× +1.07× +ToDouble + + + + + + + + +0.0104× +0.0453× +releases, oldest to newest: 1.9.0 → 2.0.0 → 2.0.1 → 2.0.2 → 2.0.3 → 2.0.4 +Measured on Intel Xeon Processor 2.80GHz. Full tables: PreciseNumber.Benchmarks. + diff --git a/scripts/benchmark_history.py b/scripts/benchmark_history.py new file mode 100644 index 0000000..a682035 --- /dev/null +++ b/scripts/benchmark_history.py @@ -0,0 +1,462 @@ +#!/usr/bin/env python3 +"""Accumulate benchmark results per release and draw them for the README. + +Two subcommands: + + ingest read BenchmarkDotNet's JSON reports and append one entry to the history file + render draw the history as SVG, one file per colour scheme + +The history file is committed, so a release only ever measures itself and the chart keeps +everything measured before it. Nothing here imports a third-party package: CI runs it on a stock +runner with no pip install step, and the SVG it writes is text that reviews like source. +""" + +from __future__ import annotations + +import argparse +import glob +import json +import os +import re +import sys +from datetime import datetime, timezone + +SCHEMA_VERSION = 1 + +# The benchmarks drawn in the README, in the order they appear. Everything measured is stored; +# this only decides what the picture shows, so it can change without re-running anything. +HEADLINE = [ + ("ArithmeticBenchmarks.Add", "30", "Add"), + ("ArithmeticBenchmarks.Multiply", "30", "Multiply"), + ("ArithmeticBenchmarks.Divide", "30", "Divide"), + ("ComparisonBenchmarks.CompareTo", "30", "CompareTo"), + ("ConstructionBenchmarks.Sanitizing", "30", "Construct"), + ("TextBenchmarks.Parse", "30", "Parse"), + ("ConversionBenchmarks.ToDouble", None, "ToDouble"), +] + +BASELINE_KEY = "BaselineBenchmarks.ReferenceWork" + +# Validated against scripts/validate_palette.js in both modes: every check passes, worst adjacent +# CVD dE 24.7 light / 26.8 dark. +THEMES = { + "light": { + "surface": "#fcfcfb", + "ink": "#0b0b0b", + "muted": "#52514e", + "grid": "#e4e3df", + "alloc": "#2a78d6", + "time": "#eb6834", + }, + "dark": { + "surface": "#1a1a19", + "ink": "#ffffff", + "muted": "#c3c2b7", + "grid": "#333330", + "alloc": "#3987e5", + "time": "#d95926", + }, +} + + +# --------------------------------------------------------------------------- ingest + + +def _benchmark_key(full_name: str) -> str: + """Reduces a fully qualified benchmark name to Class.Method.""" + bare = full_name.split("(", 1)[0] + parts = bare.split(".") + return ".".join(parts[-2:]) if len(parts) >= 2 else bare + + +def _params(report: dict) -> str: + """The parameter values for one case, as BenchmarkDotNet writes them.""" + return (report.get("Parameters") or "").strip() + + +def read_reports(results_dir: str) -> dict: + """Reads every BenchmarkDotNet JSON report below a directory.""" + measured: dict[str, dict] = {} + host = {} + paths = sorted(glob.glob(os.path.join(results_dir, "**", "*-report-full.json"), recursive=True)) + if not paths: + raise SystemExit(f"No *-report-full.json under {results_dir}") + + for path in paths: + with open(path, encoding="utf-8-sig") as handle: + document = json.load(handle) + + environment = document.get("HostEnvironmentInfo") or {} + if not host: + host = { + "cpu": (environment.get("ProcessorName") or "").strip(), + "runtime": (environment.get("RuntimeVersion") or "").strip(), + "dotnetSdk": (environment.get("DotNetSdkVersion") or "").strip(), + } + + for report in document.get("Benchmarks") or []: + statistics = report.get("Statistics") or {} + mean = statistics.get("Mean") + if mean is None: + continue + memory = report.get("Memory") or {} + key = _benchmark_key(report.get("FullName") or report.get("MethodTitle") or "") + parameters = _params(report) + measured.setdefault(key, {})[parameters] = { + "meanNs": round(float(mean), 4), + "allocatedBytes": int(memory.get("BytesAllocatedPerOperation") or 0), + } + + return {"host": host, "benchmarks": measured} + + +def ingest(args: argparse.Namespace) -> None: + gathered = read_reports(args.results) + measured = gathered["benchmarks"] + + baseline_ns = None + if BASELINE_KEY in measured: + baseline_ns = next(iter(measured[BASELINE_KEY].values()))["meanNs"] + elif args.baseline_ns is not None: + baseline_ns = args.baseline_ns + + if baseline_ns is None: + print( + f"warning: no {BASELINE_KEY} measurement and no --baseline-ns; " + "this entry's times will not be comparable across runners", + file=sys.stderr, + ) + + entry = { + "version": args.version, + "commit": args.commit, + "date": args.date or datetime.now(timezone.utc).strftime("%Y-%m-%d"), + "cpu": gathered["host"].get("cpu", ""), + "runtime": gathered["host"].get("runtime", ""), + "baselineNs": baseline_ns, + "runId": args.run_id or "", + "benchmarks": { + key: values for key, values in sorted(measured.items()) if key != BASELINE_KEY + }, + } + + history = load_history(args.history) + # A version is measured once. Re-running a release replaces its entry rather than doubling it. + history["entries"] = [e for e in history["entries"] if e.get("version") != entry["version"]] + history["entries"].append(entry) + history["entries"].sort(key=version_sort_key) + + os.makedirs(os.path.dirname(args.history) or ".", exist_ok=True) + with open(args.history, "w", encoding="utf-8", newline="\n") as handle: + json.dump(history, handle, indent=2, sort_keys=False) + handle.write("\n") + + print( + f"ingested {entry['version']}: {len(entry['benchmarks'])} benchmarks, " + f"baseline {baseline_ns} ns, cpu {entry['cpu'] or 'unknown'}" + ) + + +def load_history(path: str) -> dict: + if not os.path.exists(path): + return {"schemaVersion": SCHEMA_VERSION, "entries": []} + with open(path, encoding="utf-8") as handle: + history = json.load(handle) + history.setdefault("schemaVersion", SCHEMA_VERSION) + history.setdefault("entries", []) + return history + + +def version_sort_key(entry: dict): + """Orders versions numerically, keeping anything unparseable at the front in name order.""" + text = str(entry.get("version", "")) + numbers = [int(part) for part in re.findall(r"\d+", text)] + return (1, numbers, text) if numbers else (0, [], text) + + +# --------------------------------------------------------------------------- render + + +def series_for(history: dict, key: str, parameters: str | None): + """Pulls one benchmark's points out of every entry, in release order.""" + points = [] + for entry in history["entries"]: + cases = (entry.get("benchmarks") or {}).get(key) + if not cases: + points.append(None) + continue + if parameters is None: + case = next(iter(cases.values())) + else: + case = next( + (value for name, value in cases.items() if parameters in name), + None, + ) + points.append(case) + return points + + +def nice_bytes(value: float) -> str: + if value <= 0: + return "0 B" + if value >= 1024: + return f"{value / 1024:.1f} KB" + return f"{value:.0f} B" + + +def ratio_label(value: float) -> str: + """Three significant figures, so a 0.0331x and a 15.3x are both legible.""" + if value <= 0: + return "0×" + if value >= 100: + return f"{value:.0f}×" + if value >= 10: + return f"{value:.1f}×" + if value >= 1: + return f"{value:.2f}×" + if value >= 0.1: + return f"{value:.3f}×" + return f"{value:.4f}×" + + +def escape(text: str) -> str: + return ( + str(text) + .replace("&", "&") + .replace("<", "<") + .replace(">", ">") + .replace('"', """) + ) + + +def panel(x0, y0, width, height, title, labels, values, fmt, colour, theme): + """One small multiple: a single series, so colour carries no identity of its own.""" + out = [] + plot_top = y0 + 22 + plot_bottom = y0 + height - 20 + plot_left = x0 + 6 + plot_right = x0 + width - 10 + + out.append( + f'{escape(title)}' + ) + + present = [(i, v) for i, v in enumerate(values) if v is not None] + if not present: + out.append( + f'not measured' + ) + return out + + highest = max(v for _, v in present) + # Zero-based: these are magnitudes, and a clipped axis would exaggerate every wobble. + top = highest * 1.25 if highest > 0 else 1.0 + + def px(index): + if len(labels) == 1: + return (plot_left + plot_right) / 2 + return plot_left + (plot_right - plot_left) * index / (len(labels) - 1) + + def py(value): + return plot_bottom - (plot_bottom - plot_top) * (value / top) + + out.append( + f'' + ) + + segments = [] + for index, value in present: + segments.append(f"{px(index):.1f},{py(value):.1f}") + if len(segments) > 1: + out.append( + f'' + ) + + for index, value in present: + # A 2px surface ring keeps markers legible where the line passes behind them. + out.append( + f'' + ) + + last_index, last_value = present[-1] + anchor = "end" if last_index == len(labels) - 1 else "middle" + out.append( + f'{escape(fmt(last_value))}' + ) + + first_index, first_value = present[0] + if first_index != last_index: + out.append( + f'{escape(fmt(first_value))}' + ) + + return out + + +def render_svg(history: dict, theme_name: str) -> str: + theme = THEMES[theme_name] + entries = history["entries"] + labels = [e.get("version", "?") for e in entries] + + columns, cell_w, cell_h = 4, 228, 132 + left, right = 56, 24 + width = left + columns * cell_w + right + rows = (len(HEADLINE) + columns - 1) // columns + section_h = 34 + rows * cell_h + height = 72 + section_h * 2 + 54 + + parts = [ + f'', + "", + f'', + f'PreciseNumber performance by release', + ] + + latest = entries[-1] if entries else {} + parts.append( + f'' + f'{escape(len(entries))} releases · newest {escape(latest.get("version", "?"))}' + f'{" · " + escape(latest.get("date", "")) if latest.get("date") else ""}' + ) + + sections = [ + ( + "Allocated bytes per operation", + theme["alloc"], + lambda case: float(case["allocatedBytes"]), + nice_bytes, + "Deterministic: the same code allocates the same bytes on any machine.", + ), + ( + "Time, as a multiple of a fixed reference workload", + theme["time"], + None, # filled in below; needs the entry's baseline + ratio_label, + "Divided by a reference loop measured in the same job, which cancels most of the " + "difference between CI runners. Lower is faster.", + ), + ] + + y = 72 + for title, colour, _value_of, fmt, note in sections: + is_time = colour == theme["time"] + parts.append(f'') + parts.append(f'{escape(title)}') + parts.append(f'{escape(note)}') + + grid_top = y + 26 + for position, (key, parameters, label) in enumerate(HEADLINE): + cases = series_for(history, key, parameters) + if is_time: + values = [ + (case["meanNs"] / entry["baselineNs"]) + if case and entry.get("baselineNs") + else None + for case, entry in zip(cases, entries) + ] + else: + values = [float(case["allocatedBytes"]) if case else None for case in cases] + + column, row = position % columns, position // columns + parts.extend( + panel( + left + column * cell_w, + grid_top + row * cell_h, + cell_w - 16, + cell_h - 12, + label + ("" if parameters is None else f" ({parameters.split('=')[-1]} digits)"), + labels, + values, + fmt, + colour, + theme, + ) + ) + + y += 34 + rows * cell_h + + # One shared x axis caption: every panel uses the same release order. + axis_y = y - 4 + ticks = [] + for index, label in enumerate(labels): + if len(labels) > 6 and 0 < index < len(labels) - 1 and index % 2: + continue + ticks.append(escape(label)) + parts.append( + f'releases, oldest to newest: ' + f'{escape(" → ".join(ticks))}' + ) + + cpus = sorted({e.get("cpu", "") for e in entries if e.get("cpu")}) + parts.append( + f'' + f'Measured on {escape(", ".join(cpus) or "an unrecorded CPU")}. ' + f'Full tables: PreciseNumber.Benchmarks.' + ) + parts.append("") + return "\n".join(parts) + "\n" + + +def render(args: argparse.Namespace) -> None: + history = load_history(args.history) + if not history["entries"]: + raise SystemExit(f"{args.history} has no entries to draw") + + base, extension = os.path.splitext(args.out) + written = [] + for theme_name in ("light", "dark"): + path = args.out if theme_name == "light" else f"{base}-dark{extension}" + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + with open(path, "w", encoding="utf-8", newline="\n") as handle: + handle.write(render_svg(history, theme_name)) + written.append(path) + print(f"rendered {len(history['entries'])} releases to {', '.join(written)}") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + + take = sub.add_parser("ingest", help="append one release's results to the history") + take.add_argument("--history", required=True) + take.add_argument("--results", required=True, help="directory holding BenchmarkDotNet output") + take.add_argument("--version", required=True) + take.add_argument("--commit", default="") + take.add_argument("--date", default="") + take.add_argument("--run-id", default="") + take.add_argument( + "--baseline-ns", + type=float, + default=None, + help="reference time to record when this version predates BaselineBenchmarks", + ) + take.set_defaults(func=ingest) + + draw = sub.add_parser("render", help="draw the history as SVG") + draw.add_argument("--history", required=True) + draw.add_argument("--out", required=True, help="light-mode path; the dark file sits beside it") + draw.set_defaults(func=render) + + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() From 01928f3c86118b70c4feb12f252db209dfe60164 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 03:34:08 +0000 Subject: [PATCH 2/4] Write the benchmark history tool in C# rather than Python [patch] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tool that reads BenchmarkDotNet's reports and draws the chart was Python, which fits none of these repositories: there is no other .py file in PreciseNumber, Semantics or SignificantNumber, and what scripting they do have is PowerShell. It was chosen because it was quick to write and needs nothing installed on a runner, which is a reason about writing it rather than about living with it — and it was about to be copied into two more repositories. It is now a .NET 10 file-based app, run as dotnet run scripts/benchmark-history.cs -- ingest ... so it needs no project, no runtime beyond the SDK that already builds the library, and no second language in the workflow. A third subcommand, baseline, replaces the inline Python that read one number back out of a report. The build covers it, which is the part that could not be had before: the first compile failed on CA1305, CA1502, CA1505, CA1506, IL2026, IL3050 and SYSLIB1045. Most are fixed rather than suppressed — cultures are explicit, the regex is source-generated, and the work moved out of top-level statements into a class so the analyzers judge each method rather than one 500-line Main. The port is verified by output rather than by reading: re-ingesting the same reports and re-rendering leaves both the history and the two SVGs byte for byte identical to what Python produced. Getting there caught one real difference — parameters were coming out in ordinal order, which reads 200, 30, 8, where BenchmarkDotNet declares them 8, 30, 200 and the Digits axis is meant to be read across in order. Also drops the .gitignore entries added with the workflow. ktsu.Sdk regenerates that file on every build, so they did not survive one — which is what quietly ate them twice while this was being written. The scratch output goes to BenchmarkDotNet.Artifacts/ instead, already ignored, and worktrees are created under RUNNER_TEMP, outside the repository where the commit step cannot see them. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017jrnV7N94UGL8fDRRE8Xt8 --- .github/workflows/benchmark-history.yml | 33 +- scripts/benchmark-history.cs | 601 ++++++++++++++++++++++++ scripts/benchmark_history.py | 462 ------------------ 3 files changed, 615 insertions(+), 481 deletions(-) create mode 100644 scripts/benchmark-history.cs delete mode 100644 scripts/benchmark_history.py diff --git a/.github/workflows/benchmark-history.yml b/.github/workflows/benchmark-history.yml index 29e508a..4c90f7d 100644 --- a/.github/workflows/benchmark-history.yml +++ b/.github/workflows/benchmark-history.yml @@ -39,6 +39,8 @@ concurrency: env: DOTNET_VERSION: "10.0" HISTORY: docs/benchmarks/history.json + # Already ignored, and ktsu.Sdk regenerates .gitignore on build so a new entry would not last. + RUNS: BenchmarkDotNet.Artifacts CHART: docs/benchmarks/performance.svg # The set drawn in the README. Name globs rather than a [BenchmarkCategory], because the backfill # runs this same filter against older checkouts that predate any attribute added today. @@ -87,15 +89,8 @@ jobs: dotnet run -c Release --project PreciseNumber.Benchmarks -- \ --filter '*BaselineBenchmarks.ReferenceWork' \ --job "$BENCHMARK_JOB" \ - --artifacts "$GITHUB_WORKSPACE/bench/baseline" - ns=$(python3 - <<'PY' - import glob, json - path = glob.glob("bench/baseline/results/*BaselineBenchmarks-report-full.json")[0] - with open(path, encoding="utf-8-sig") as handle: - document = json.load(handle) - print(round(document["Benchmarks"][0]["Statistics"]["Mean"], 4)) - PY - ) + --artifacts "$GITHUB_WORKSPACE/$RUNS/baseline" + ns=$(dotnet run scripts/benchmark-history.cs -- baseline --results "$RUNS/baseline") echo "Reference workload: $ns ns" echo "ns=$ns" >> "$GITHUB_OUTPUT" @@ -107,17 +102,17 @@ jobs: run: | set -euo pipefail version="${VERSION#v}" - work="$GITHUB_WORKSPACE/backfill/$version" + work="${RUNNER_TEMP}/bench-$version" git worktree add --detach "$work" "$VERSION" (cd "$work" && dotnet run -c Release --project PreciseNumber.Benchmarks -- \ --filter $HEADLINE_FILTER \ --job "$BENCHMARK_JOB" \ - --artifacts "$GITHUB_WORKSPACE/bench/$version") + --artifacts "$GITHUB_WORKSPACE/$RUNS/$version") - python3 scripts/benchmark_history.py ingest \ + dotnet run scripts/benchmark-history.cs -- ingest \ --history "$HISTORY" \ - --results "bench/$version" \ + --results "$RUNS/$version" \ --version "$version" \ --commit "$(git rev-parse --short "$VERSION^{commit}")" \ --date "$(git log -1 --format=%cs "$VERSION")" \ @@ -141,7 +136,7 @@ jobs: for index in "${!refs[@]}"; do ref="${refs[$index]}" label="${labels[$index]:-${ref#v}}" - work="$GITHUB_WORKSPACE/backfill/$label" + work="${RUNNER_TEMP}/bench-$label" echo "::group::$label ($ref)" rm -rf "$work" @@ -159,11 +154,11 @@ jobs: (cd "$work" && dotnet run -c Release --project PreciseNumber.Benchmarks -- \ --filter $HEADLINE_FILTER \ --job "$BENCHMARK_JOB" \ - --artifacts "$GITHUB_WORKSPACE/bench/$label") + --artifacts "$GITHUB_WORKSPACE/$RUNS/$label") - python3 scripts/benchmark_history.py ingest \ + dotnet run scripts/benchmark-history.cs -- ingest \ --history "$HISTORY" \ - --results "bench/$label" \ + --results "$RUNS/$label" \ --version "$label" \ --commit "$(git rev-parse --short "$ref^{commit}")" \ --date "$(git log -1 --format=%cs "$ref")" \ @@ -176,7 +171,7 @@ jobs: - name: Redraw the chart shell: bash - run: python3 scripts/benchmark_history.py render --history "$HISTORY" --out "$CHART" + run: dotnet run scripts/benchmark-history.cs -- render --history "$HISTORY" --out "$CHART" - name: Commit the history and the chart shell: bash @@ -204,6 +199,6 @@ jobs: uses: actions/upload-artifact@v7 with: name: benchmark-history-${{ github.run_id }} - path: bench/ + path: ${{ env.RUNS }}/ retention-days: 30 if-no-files-found: warn diff --git a/scripts/benchmark-history.cs b/scripts/benchmark-history.cs new file mode 100644 index 0000000..c96c609 --- /dev/null +++ b/scripts/benchmark-history.cs @@ -0,0 +1,601 @@ +// Copyright (c) 2023-2026 ktsu-dev contributors + +// Accumulates benchmark results per release and draws them for the README. +// +// dotnet run scripts/benchmark-history.cs -- ingest --history --results --version +// dotnet run scripts/benchmark-history.cs -- render --history --out +// +// A file-based app rather than a project: it is tooling, it is the same language as the library, +// and the SDK that builds the library already runs it with nothing else installed. The work still +// lives in a class rather than in top-level statements, so the analyzers judge each method. + +using System.Globalization; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Text.RegularExpressions; + +return BenchmarkHistory.Run(args); + +/// Reads BenchmarkDotNet reports into a per-release history, and draws it. +internal static partial class BenchmarkHistory +{ + private const int SchemaVersion = 1; + private const string BaselineKey = "BaselineBenchmarks.ReferenceWork"; + private const int Columns = 4; + private const int CellWidth = 228; + private const int CellHeight = 132; + private const int Left = 56; + + /// The benchmarks the README draws, in order. + /// + /// Everything measured is stored; this only decides what the picture shows, so it can change + /// without re-running anything. + /// + private static readonly (string Key, string? Parameters, string Label)[] Headline = + [ + ("ArithmeticBenchmarks.Add", "30", "Add"), + ("ArithmeticBenchmarks.Multiply", "30", "Multiply"), + ("ArithmeticBenchmarks.Divide", "30", "Divide"), + ("ComparisonBenchmarks.CompareTo", "30", "CompareTo"), + ("ConstructionBenchmarks.Sanitizing", "30", "Construct"), + ("TextBenchmarks.Parse", "30", "Parse"), + ("ConversionBenchmarks.ToDouble", null, "ToDouble"), + ]; + + /// + /// Validated for colour-vision separation against both surfaces: every check passes, worst + /// adjacent pair dE 24.7 light and 26.8 dark. + /// + private static readonly Dictionary Themes = new(StringComparer.Ordinal) + { + ["light"] = new("#fcfcfb", "#0b0b0b", "#52514e", "#e4e3df", "#2a78d6", "#eb6834"), + ["dark"] = new("#1a1a19", "#ffffff", "#c3c2b7", "#333330", "#3987e5", "#d95926"), + }; + + private sealed record Theme( + string Surface, string Ink, string Muted, string Grid, string Alloc, string Time); + + internal static int Run(string[] args) + { + if (args.Length == 0) + { + Console.Error.WriteLine("Expected 'ingest', 'render', or 'baseline'."); + return 2; + } + + Dictionary options = ReadOptions(args.Skip(1)); + try + { + return args[0] switch + { + "ingest" => Ingest(options), + "render" => Render(options), + "baseline" => PrintBaseline(options), + _ => Unknown(args[0]), + }; + } + catch (InvalidOperationException problem) + { + Console.Error.WriteLine(problem.Message); + return 2; + } + } + + /// Prints the reference workload's mean, for a workflow to carry between steps. + private static int PrintBaseline(Dictionary options) + { + string directory = Required(options, "results"); + string[] reports = Directory.GetFiles(directory, "*-report-full.json", SearchOption.AllDirectories); + if (reports.Length == 0) + { + Console.Error.WriteLine($"No *-report-full.json under {directory}"); + return 1; + } + + (var measured, _, _) = ReadReports(reports); + double? baseline = Baseline(measured, ""); + if (baseline is null) + { + Console.Error.WriteLine($"No {BaselineKey} measurement under {directory}"); + return 1; + } + + Console.WriteLine(baseline.Value.ToString(CultureInfo.InvariantCulture)); + return 0; + } + + private static int Unknown(string command) + { + Console.Error.WriteLine($"Unknown command '{command}'."); + return 2; + } + + private static Dictionary ReadOptions(IEnumerable rest) + { + Dictionary found = new(StringComparer.Ordinal); + string? name = null; + foreach (string argument in rest) + { + if (argument.StartsWith("--", StringComparison.Ordinal)) + { + name = argument[2..]; + found[name] = ""; + } + else if (name is not null) + { + found[name] = argument; + name = null; + } + } + + return found; + } + + private static string Required(Dictionary options, string name) => + options.TryGetValue(name, out string? value) && value.Length > 0 + ? value + : throw new InvalidOperationException($"--{name} is required"); + + private static string Optional(Dictionary options, string name, string fallback = "") => + options.TryGetValue(name, out string? value) && value.Length > 0 ? value : fallback; + + private static int Ingest(Dictionary options) + { + string resultsDirectory = Required(options, "results"); + string[] reports = Directory.GetFiles(resultsDirectory, "*-report-full.json", SearchOption.AllDirectories); + Array.Sort(reports, StringComparer.Ordinal); + if (reports.Length == 0) + { + Console.Error.WriteLine($"No *-report-full.json under {resultsDirectory}"); + return 1; + } + + (var measured, string cpu, string runtime) = ReadReports(reports); + double? baseline = Baseline(measured, Optional(options, "baseline-ns")); + if (baseline is null) + { + Console.Error.WriteLine( + $"warning: no {BaselineKey} measurement and no --baseline-ns; " + + "this entry's times will not be comparable across runners"); + } + + string version = Required(options, "version"); + JsonObject record = new() + { + ["version"] = version, + ["commit"] = Optional(options, "commit"), + ["date"] = Optional(options, "date", DateTime.UtcNow.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)), + ["cpu"] = cpu, + ["runtime"] = runtime, + ["baselineNs"] = baseline, + ["runId"] = Optional(options, "run-id"), + ["benchmarks"] = Benchmarks(measured), + }; + + string historyPath = Required(options, "history"); + JsonObject history = LoadHistory(historyPath); + JsonArray entries = history["entries"]!.AsArray(); + + // A version is measured once. Re-running a release replaces its entry rather than doubling it. + for (int index = entries.Count - 1; index >= 0; index--) + { + if (string.Equals(entries[index]?["version"]?.GetValue(), version, StringComparison.Ordinal)) + { + entries.RemoveAt(index); + } + } + + entries.Add((JsonNode?)record); + Reorder(entries); + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(historyPath))!); + File.WriteAllText(historyPath, history.ToJsonString(new JsonSerializerOptions { WriteIndented = true }) + "\n"); + + Console.WriteLine( + $"ingested {version}: {record["benchmarks"]!.AsObject().Count} benchmarks, " + + $"baseline {baseline?.ToString(CultureInfo.InvariantCulture) ?? "none"} ns, " + + $"cpu {(cpu.Length > 0 ? cpu : "unknown")}"); + return 0; + } + + private static (SortedDictionary> Measured, string Cpu, string Runtime) + ReadReports(string[] reports) + { + SortedDictionary> measured = new(StringComparer.Ordinal); + string cpu = ""; + string runtime = ""; + + foreach (string report in reports) + { + JsonNode document = JsonNode.Parse(File.ReadAllText(report))!; + if (cpu.Length == 0 && document["HostEnvironmentInfo"] is JsonNode environment) + { + cpu = (environment["ProcessorName"]?.GetValue() ?? "").Trim(); + runtime = (environment["RuntimeVersion"]?.GetValue() ?? "").Trim(); + } + + foreach (JsonNode? entry in document["Benchmarks"]?.AsArray() ?? []) + { + if (entry?["Statistics"]?["Mean"] is not JsonNode mean) + { + continue; + } + + string key = BenchmarkKey(entry["FullName"]?.GetValue() ?? ""); + string parameters = (entry["Parameters"]?.GetValue() ?? "").Trim(); + if (!measured.TryGetValue(key, out List? cases)) + { + cases = []; + measured[key] = cases; + } + + Measurement measurement = new( + Math.Round(mean.GetValue(), 4), + entry["Memory"]?["BytesAllocatedPerOperation"]?.GetValue() ?? 0); + int existing = cases.FindIndex(one => string.Equals(one.Parameters, parameters, StringComparison.Ordinal)); + if (existing >= 0) + { + cases[existing] = new(parameters, measurement); + } + else + { + cases.Add(new(parameters, measurement)); + } + } + } + + return (measured, cpu, runtime); + } + + private sealed record Measurement(double MeanNs, long AllocatedBytes); + + private sealed record ParameterCase(string Parameters, Measurement Value); + + private static double? Baseline( + SortedDictionary> measured, string given) + { + if (measured.TryGetValue(BaselineKey, out List? cases) && cases.Count > 0) + { + return cases[0].Value.MeanNs; + } + + return double.TryParse(given, NumberStyles.Float, CultureInfo.InvariantCulture, out double parsed) + ? parsed + : null; + } + + private static JsonObject Benchmarks( + SortedDictionary> measured) + { + JsonObject benchmarks = []; + foreach ((string key, List cases) in measured) + { + if (string.Equals(key, BaselineKey, StringComparison.Ordinal)) + { + continue; + } + + JsonObject byParameters = []; + foreach (ParameterCase one in cases) + { + byParameters[one.Parameters] = new JsonObject + { + ["meanNs"] = one.Value.MeanNs, + ["allocatedBytes"] = one.Value.AllocatedBytes, + }; + } + + benchmarks[key] = byParameters; + } + + return benchmarks; + } + + private static void Reorder(JsonArray entries) + { + JsonNode[] ordered = + [ + .. entries + .Select(node => node!.DeepClone()) + .OrderBy(node => node["version"]?.GetValue() ?? "", VersionOrder.Instance), + ]; + + entries.Clear(); + foreach (JsonNode node in ordered) + { + entries.Add((JsonNode?)node); + } + } + + private static string BenchmarkKey(string fullName) + { + string bare = fullName.Split('(')[0]; + string[] parts = bare.Split('.'); + return parts.Length >= 2 ? $"{parts[^2]}.{parts[^1]}" : bare; + } + + private static JsonObject LoadHistory(string path) + { + if (!File.Exists(path)) + { + return new JsonObject { ["schemaVersion"] = SchemaVersion, ["entries"] = new JsonArray() }; + } + + JsonObject history = JsonNode.Parse(File.ReadAllText(path))!.AsObject(); + history["schemaVersion"] ??= SchemaVersion; + history["entries"] ??= new JsonArray(); + return history; + } + + /// Orders versions numerically, keeping anything unparseable first in name order. + private sealed class VersionOrder : IComparer + { + internal static readonly VersionOrder Instance = new(); + + public int Compare(string? left, string? right) + { + int[] first = Numbers(left); + int[] second = Numbers(right); + for (int index = 0; index < Math.Min(first.Length, second.Length); index++) + { + if (first[index] != second[index]) + { + return first[index].CompareTo(second[index]); + } + } + + return first.Length != second.Length + ? first.Length.CompareTo(second.Length) + : string.CompareOrdinal(left, right); + } + + private static int[] Numbers(string? text) => + [.. DigitRun().Matches(text ?? "").Select(match => int.Parse(match.Value, CultureInfo.InvariantCulture))]; + } + + [GeneratedRegex("[0-9]+")] + private static partial Regex DigitRun(); + + private static int Render(Dictionary options) + { + string historyPath = Required(options, "history"); + JsonArray entries = LoadHistory(historyPath)["entries"]!.AsArray(); + if (entries.Count == 0) + { + Console.Error.WriteLine($"{historyPath} has no entries to draw"); + return 1; + } + + string output = Required(options, "out"); + string extension = Path.GetExtension(output); + string stem = output[..^extension.Length]; + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(output))!); + + List written = []; + foreach (string name in (string[])["light", "dark"]) + { + string path = string.Equals(name, "light", StringComparison.Ordinal) + ? output + : $"{stem}-dark{extension}"; + File.WriteAllText(path, Draw(entries, Themes[name])); + written.Add(path); + } + + Console.WriteLine($"rendered {entries.Count} releases to {string.Join(", ", written)}"); + return 0; + } + + private static string Draw(JsonArray entries, Theme theme) + { + string[] labels = [.. entries.Select(entry => entry!["version"]?.GetValue() ?? "?")]; + int width = Left + (Columns * CellWidth) + 24; + int rows = (Headline.Length + Columns - 1) / Columns; + int height = 72 + (((34 + (rows * CellHeight)) * 2) + 54); + + StringBuilder svg = new(); + Preamble(svg, theme, width, height, entries); + + int y = 72; + foreach (bool isTime in (bool[])[false, true]) + { + Section(svg, theme, entries, labels.Length, y, isTime); + y += 34 + (rows * CellHeight); + } + + Footer(svg, entries, labels, y - 4); + svg.AppendLine(""); + return svg.ToString(); + } + + private static void Preamble(StringBuilder svg, Theme theme, int width, int height, JsonArray entries) + { + svg.AppendLine(CultureInfo.InvariantCulture, $""""""); + svg.AppendLine(""); + svg.AppendLine(CultureInfo.InvariantCulture, $""""""); + svg.AppendLine(CultureInfo.InvariantCulture, $"""PreciseNumber performance by release"""); + + JsonNode latest = entries[^1]!; + string date = latest["date"]?.GetValue() ?? ""; + string suffix = date.Length > 0 ? " · " + Escape(date) : ""; + svg.AppendLine(CultureInfo.InvariantCulture, $"""{entries.Count} releases · newest {Escape(latest["version"]?.GetValue() ?? "?")}{suffix}"""); + } + + private static void Section(StringBuilder svg, Theme theme, JsonArray entries, int points, int y, bool isTime) + { + string colour = isTime ? theme.Time : theme.Alloc; + string title = isTime + ? "Time, as a multiple of a fixed reference workload" + : "Allocated bytes per operation"; + string note = isTime + ? "Divided by a reference loop measured in the same job, which cancels most of the difference between CI runners. Lower is faster." + : "Deterministic: the same code allocates the same bytes on any machine."; + + svg.AppendLine(CultureInfo.InvariantCulture, $""""""); + svg.AppendLine(CultureInfo.InvariantCulture, $"""{Escape(title)}"""); + svg.AppendLine(CultureInfo.InvariantCulture, $"""{Escape(note)}"""); + + for (int position = 0; position < Headline.Length; position++) + { + (string key, string? parameters, string label) = Headline[position]; + double?[] values = [.. entries.Select(entry => Value(entry!, key, parameters, isTime))]; + Panel( + svg, + Left + (position % Columns * CellWidth), + y + 26 + (position / Columns * CellHeight), + label + (parameters is null ? "" : $" ({parameters} digits)"), + points, + values, + isTime, + colour, + theme); + } + } + + private static double? Value(JsonNode entry, string key, string? parameters, bool isTime) + { + if (entry["benchmarks"]?[key] is not JsonObject cases || cases.Count == 0) + { + return null; + } + + JsonNode? measurement = parameters is null + ? cases.First().Value + : cases.FirstOrDefault(pair => pair.Key.Contains(parameters, StringComparison.Ordinal)).Value; + if (measurement is null) + { + return null; + } + + if (!isTime) + { + return measurement["allocatedBytes"]!.GetValue(); + } + + double? baseline = entry["baselineNs"]?.GetValue(); + return baseline is > 0 ? measurement["meanNs"]!.GetValue() / baseline : null; + } + + private static void Footer(StringBuilder svg, JsonArray entries, string[] labels, int axisY) + { + List ticks = []; + for (int index = 0; index < labels.Length; index++) + { + if (labels.Length > 6 && index > 0 && index < labels.Length - 1 && index % 2 == 1) + { + continue; + } + + ticks.Add(Escape(labels[index])); + } + + svg.AppendLine(CultureInfo.InvariantCulture, $"""releases, oldest to newest: {string.Join(" → ", ticks)}"""); + + string[] cpus = + [ + .. entries + .Select(entry => entry!["cpu"]?.GetValue() ?? "") + .Where(name => name.Length > 0) + .Distinct(StringComparer.Ordinal) + .OrderBy(name => name, StringComparer.Ordinal), + ]; + string measured = cpus.Length > 0 ? string.Join(", ", cpus) : "an unrecorded CPU"; + svg.AppendLine(CultureInfo.InvariantCulture, $"""Measured on {Escape(measured)}. Full tables: PreciseNumber.Benchmarks."""); + } + + /// One small multiple: a single series, so colour carries no identity of its own. + private static void Panel( + StringBuilder svg, int x0, int y0, string title, int points, double?[] values, + bool isTime, string colour, Theme theme) + { + double plotTop = y0 + 22; + double plotBottom = y0 + CellHeight - 12 - 20; + double plotLeft = x0 + 6; + double plotRight = x0 + CellWidth - 16 - 10; + + svg.AppendLine(CultureInfo.InvariantCulture, $"""{Escape(title)}"""); + + (int Index, double Value)[] present = + [ + .. values + .Select((value, index) => (Index: index, Value: value)) + .Where(point => point.Value.HasValue) + .Select(point => (point.Index, point.Value!.Value)), + ]; + + if (present.Length == 0) + { + svg.AppendLine(CultureInfo.InvariantCulture, $"""not measured"""); + return; + } + + // Zero-based: these are magnitudes, and a clipped axis would exaggerate every wobble. + double highest = present.Max(point => point.Value); + double top = highest > 0 ? highest * 1.25 : 1.0; + + double X(int index) => points == 1 + ? (plotLeft + plotRight) / 2 + : plotLeft + ((plotRight - plotLeft) * index / (points - 1)); + double Y(double value) => plotBottom - ((plotBottom - plotTop) * (value / top)); + + svg.AppendLine(CultureInfo.InvariantCulture, $""""""); + + if (present.Length > 1) + { + string line = string.Join(" ", present.Select(point => $"{F(X(point.Index))},{F(Y(point.Value))}")); + svg.AppendLine(CultureInfo.InvariantCulture, $""""""); + } + + foreach ((int index, double value) in present) + { + // A 2px surface ring keeps markers legible where the line passes behind them. + svg.AppendLine(CultureInfo.InvariantCulture, $""""""); + } + + (int lastIndex, double lastValue) = present[^1]; + string anchor = lastIndex == points - 1 ? "end" : "middle"; + svg.AppendLine(CultureInfo.InvariantCulture, $"""{Escape(Label(lastValue, isTime))}"""); + + (int firstIndex, double firstValue) = present[0]; + if (firstIndex != lastIndex) + { + svg.AppendLine(CultureInfo.InvariantCulture, $"""{Escape(Label(firstValue, isTime))}"""); + } + } + + /// One decimal is ample for an SVG coordinate, and keeps the committed diff small. + private static string F(double value) => value.ToString("0.0", CultureInfo.InvariantCulture); + + private static string Label(double value, bool isTime) => + isTime ? RatioLabel(value) : ByteLabel(value); + + private static string ByteLabel(double value) => + value <= 0 ? "0 B" + : value >= 1024 ? (value / 1024).ToString("0.0", CultureInfo.InvariantCulture) + " KB" + : value.ToString("0", CultureInfo.InvariantCulture) + " B"; + + /// Three significant figures, so a 0.0331x and a 15.3x are both legible. + private static string RatioLabel(double value) => + value <= 0 ? "0×" + : value >= 100 ? value.ToString("0", CultureInfo.InvariantCulture) + "×" + : value >= 10 ? value.ToString("0.0", CultureInfo.InvariantCulture) + "×" + : value >= 1 ? value.ToString("0.00", CultureInfo.InvariantCulture) + "×" + : value >= 0.1 ? value.ToString("0.000", CultureInfo.InvariantCulture) + "×" + : value.ToString("0.0000", CultureInfo.InvariantCulture) + "×"; + + private static string Escape(string text) => + text + .Replace("&", "&", StringComparison.Ordinal) + .Replace("<", "<", StringComparison.Ordinal) + .Replace(">", ">", StringComparison.Ordinal) + .Replace("\"", """, StringComparison.Ordinal); +} diff --git a/scripts/benchmark_history.py b/scripts/benchmark_history.py deleted file mode 100644 index a682035..0000000 --- a/scripts/benchmark_history.py +++ /dev/null @@ -1,462 +0,0 @@ -#!/usr/bin/env python3 -"""Accumulate benchmark results per release and draw them for the README. - -Two subcommands: - - ingest read BenchmarkDotNet's JSON reports and append one entry to the history file - render draw the history as SVG, one file per colour scheme - -The history file is committed, so a release only ever measures itself and the chart keeps -everything measured before it. Nothing here imports a third-party package: CI runs it on a stock -runner with no pip install step, and the SVG it writes is text that reviews like source. -""" - -from __future__ import annotations - -import argparse -import glob -import json -import os -import re -import sys -from datetime import datetime, timezone - -SCHEMA_VERSION = 1 - -# The benchmarks drawn in the README, in the order they appear. Everything measured is stored; -# this only decides what the picture shows, so it can change without re-running anything. -HEADLINE = [ - ("ArithmeticBenchmarks.Add", "30", "Add"), - ("ArithmeticBenchmarks.Multiply", "30", "Multiply"), - ("ArithmeticBenchmarks.Divide", "30", "Divide"), - ("ComparisonBenchmarks.CompareTo", "30", "CompareTo"), - ("ConstructionBenchmarks.Sanitizing", "30", "Construct"), - ("TextBenchmarks.Parse", "30", "Parse"), - ("ConversionBenchmarks.ToDouble", None, "ToDouble"), -] - -BASELINE_KEY = "BaselineBenchmarks.ReferenceWork" - -# Validated against scripts/validate_palette.js in both modes: every check passes, worst adjacent -# CVD dE 24.7 light / 26.8 dark. -THEMES = { - "light": { - "surface": "#fcfcfb", - "ink": "#0b0b0b", - "muted": "#52514e", - "grid": "#e4e3df", - "alloc": "#2a78d6", - "time": "#eb6834", - }, - "dark": { - "surface": "#1a1a19", - "ink": "#ffffff", - "muted": "#c3c2b7", - "grid": "#333330", - "alloc": "#3987e5", - "time": "#d95926", - }, -} - - -# --------------------------------------------------------------------------- ingest - - -def _benchmark_key(full_name: str) -> str: - """Reduces a fully qualified benchmark name to Class.Method.""" - bare = full_name.split("(", 1)[0] - parts = bare.split(".") - return ".".join(parts[-2:]) if len(parts) >= 2 else bare - - -def _params(report: dict) -> str: - """The parameter values for one case, as BenchmarkDotNet writes them.""" - return (report.get("Parameters") or "").strip() - - -def read_reports(results_dir: str) -> dict: - """Reads every BenchmarkDotNet JSON report below a directory.""" - measured: dict[str, dict] = {} - host = {} - paths = sorted(glob.glob(os.path.join(results_dir, "**", "*-report-full.json"), recursive=True)) - if not paths: - raise SystemExit(f"No *-report-full.json under {results_dir}") - - for path in paths: - with open(path, encoding="utf-8-sig") as handle: - document = json.load(handle) - - environment = document.get("HostEnvironmentInfo") or {} - if not host: - host = { - "cpu": (environment.get("ProcessorName") or "").strip(), - "runtime": (environment.get("RuntimeVersion") or "").strip(), - "dotnetSdk": (environment.get("DotNetSdkVersion") or "").strip(), - } - - for report in document.get("Benchmarks") or []: - statistics = report.get("Statistics") or {} - mean = statistics.get("Mean") - if mean is None: - continue - memory = report.get("Memory") or {} - key = _benchmark_key(report.get("FullName") or report.get("MethodTitle") or "") - parameters = _params(report) - measured.setdefault(key, {})[parameters] = { - "meanNs": round(float(mean), 4), - "allocatedBytes": int(memory.get("BytesAllocatedPerOperation") or 0), - } - - return {"host": host, "benchmarks": measured} - - -def ingest(args: argparse.Namespace) -> None: - gathered = read_reports(args.results) - measured = gathered["benchmarks"] - - baseline_ns = None - if BASELINE_KEY in measured: - baseline_ns = next(iter(measured[BASELINE_KEY].values()))["meanNs"] - elif args.baseline_ns is not None: - baseline_ns = args.baseline_ns - - if baseline_ns is None: - print( - f"warning: no {BASELINE_KEY} measurement and no --baseline-ns; " - "this entry's times will not be comparable across runners", - file=sys.stderr, - ) - - entry = { - "version": args.version, - "commit": args.commit, - "date": args.date or datetime.now(timezone.utc).strftime("%Y-%m-%d"), - "cpu": gathered["host"].get("cpu", ""), - "runtime": gathered["host"].get("runtime", ""), - "baselineNs": baseline_ns, - "runId": args.run_id or "", - "benchmarks": { - key: values for key, values in sorted(measured.items()) if key != BASELINE_KEY - }, - } - - history = load_history(args.history) - # A version is measured once. Re-running a release replaces its entry rather than doubling it. - history["entries"] = [e for e in history["entries"] if e.get("version") != entry["version"]] - history["entries"].append(entry) - history["entries"].sort(key=version_sort_key) - - os.makedirs(os.path.dirname(args.history) or ".", exist_ok=True) - with open(args.history, "w", encoding="utf-8", newline="\n") as handle: - json.dump(history, handle, indent=2, sort_keys=False) - handle.write("\n") - - print( - f"ingested {entry['version']}: {len(entry['benchmarks'])} benchmarks, " - f"baseline {baseline_ns} ns, cpu {entry['cpu'] or 'unknown'}" - ) - - -def load_history(path: str) -> dict: - if not os.path.exists(path): - return {"schemaVersion": SCHEMA_VERSION, "entries": []} - with open(path, encoding="utf-8") as handle: - history = json.load(handle) - history.setdefault("schemaVersion", SCHEMA_VERSION) - history.setdefault("entries", []) - return history - - -def version_sort_key(entry: dict): - """Orders versions numerically, keeping anything unparseable at the front in name order.""" - text = str(entry.get("version", "")) - numbers = [int(part) for part in re.findall(r"\d+", text)] - return (1, numbers, text) if numbers else (0, [], text) - - -# --------------------------------------------------------------------------- render - - -def series_for(history: dict, key: str, parameters: str | None): - """Pulls one benchmark's points out of every entry, in release order.""" - points = [] - for entry in history["entries"]: - cases = (entry.get("benchmarks") or {}).get(key) - if not cases: - points.append(None) - continue - if parameters is None: - case = next(iter(cases.values())) - else: - case = next( - (value for name, value in cases.items() if parameters in name), - None, - ) - points.append(case) - return points - - -def nice_bytes(value: float) -> str: - if value <= 0: - return "0 B" - if value >= 1024: - return f"{value / 1024:.1f} KB" - return f"{value:.0f} B" - - -def ratio_label(value: float) -> str: - """Three significant figures, so a 0.0331x and a 15.3x are both legible.""" - if value <= 0: - return "0×" - if value >= 100: - return f"{value:.0f}×" - if value >= 10: - return f"{value:.1f}×" - if value >= 1: - return f"{value:.2f}×" - if value >= 0.1: - return f"{value:.3f}×" - return f"{value:.4f}×" - - -def escape(text: str) -> str: - return ( - str(text) - .replace("&", "&") - .replace("<", "<") - .replace(">", ">") - .replace('"', """) - ) - - -def panel(x0, y0, width, height, title, labels, values, fmt, colour, theme): - """One small multiple: a single series, so colour carries no identity of its own.""" - out = [] - plot_top = y0 + 22 - plot_bottom = y0 + height - 20 - plot_left = x0 + 6 - plot_right = x0 + width - 10 - - out.append( - f'{escape(title)}' - ) - - present = [(i, v) for i, v in enumerate(values) if v is not None] - if not present: - out.append( - f'not measured' - ) - return out - - highest = max(v for _, v in present) - # Zero-based: these are magnitudes, and a clipped axis would exaggerate every wobble. - top = highest * 1.25 if highest > 0 else 1.0 - - def px(index): - if len(labels) == 1: - return (plot_left + plot_right) / 2 - return plot_left + (plot_right - plot_left) * index / (len(labels) - 1) - - def py(value): - return plot_bottom - (plot_bottom - plot_top) * (value / top) - - out.append( - f'' - ) - - segments = [] - for index, value in present: - segments.append(f"{px(index):.1f},{py(value):.1f}") - if len(segments) > 1: - out.append( - f'' - ) - - for index, value in present: - # A 2px surface ring keeps markers legible where the line passes behind them. - out.append( - f'' - ) - - last_index, last_value = present[-1] - anchor = "end" if last_index == len(labels) - 1 else "middle" - out.append( - f'{escape(fmt(last_value))}' - ) - - first_index, first_value = present[0] - if first_index != last_index: - out.append( - f'{escape(fmt(first_value))}' - ) - - return out - - -def render_svg(history: dict, theme_name: str) -> str: - theme = THEMES[theme_name] - entries = history["entries"] - labels = [e.get("version", "?") for e in entries] - - columns, cell_w, cell_h = 4, 228, 132 - left, right = 56, 24 - width = left + columns * cell_w + right - rows = (len(HEADLINE) + columns - 1) // columns - section_h = 34 + rows * cell_h - height = 72 + section_h * 2 + 54 - - parts = [ - f'', - "", - f'', - f'PreciseNumber performance by release', - ] - - latest = entries[-1] if entries else {} - parts.append( - f'' - f'{escape(len(entries))} releases · newest {escape(latest.get("version", "?"))}' - f'{" · " + escape(latest.get("date", "")) if latest.get("date") else ""}' - ) - - sections = [ - ( - "Allocated bytes per operation", - theme["alloc"], - lambda case: float(case["allocatedBytes"]), - nice_bytes, - "Deterministic: the same code allocates the same bytes on any machine.", - ), - ( - "Time, as a multiple of a fixed reference workload", - theme["time"], - None, # filled in below; needs the entry's baseline - ratio_label, - "Divided by a reference loop measured in the same job, which cancels most of the " - "difference between CI runners. Lower is faster.", - ), - ] - - y = 72 - for title, colour, _value_of, fmt, note in sections: - is_time = colour == theme["time"] - parts.append(f'') - parts.append(f'{escape(title)}') - parts.append(f'{escape(note)}') - - grid_top = y + 26 - for position, (key, parameters, label) in enumerate(HEADLINE): - cases = series_for(history, key, parameters) - if is_time: - values = [ - (case["meanNs"] / entry["baselineNs"]) - if case and entry.get("baselineNs") - else None - for case, entry in zip(cases, entries) - ] - else: - values = [float(case["allocatedBytes"]) if case else None for case in cases] - - column, row = position % columns, position // columns - parts.extend( - panel( - left + column * cell_w, - grid_top + row * cell_h, - cell_w - 16, - cell_h - 12, - label + ("" if parameters is None else f" ({parameters.split('=')[-1]} digits)"), - labels, - values, - fmt, - colour, - theme, - ) - ) - - y += 34 + rows * cell_h - - # One shared x axis caption: every panel uses the same release order. - axis_y = y - 4 - ticks = [] - for index, label in enumerate(labels): - if len(labels) > 6 and 0 < index < len(labels) - 1 and index % 2: - continue - ticks.append(escape(label)) - parts.append( - f'releases, oldest to newest: ' - f'{escape(" → ".join(ticks))}' - ) - - cpus = sorted({e.get("cpu", "") for e in entries if e.get("cpu")}) - parts.append( - f'' - f'Measured on {escape(", ".join(cpus) or "an unrecorded CPU")}. ' - f'Full tables: PreciseNumber.Benchmarks.' - ) - parts.append("") - return "\n".join(parts) + "\n" - - -def render(args: argparse.Namespace) -> None: - history = load_history(args.history) - if not history["entries"]: - raise SystemExit(f"{args.history} has no entries to draw") - - base, extension = os.path.splitext(args.out) - written = [] - for theme_name in ("light", "dark"): - path = args.out if theme_name == "light" else f"{base}-dark{extension}" - os.makedirs(os.path.dirname(path) or ".", exist_ok=True) - with open(path, "w", encoding="utf-8", newline="\n") as handle: - handle.write(render_svg(history, theme_name)) - written.append(path) - print(f"rendered {len(history['entries'])} releases to {', '.join(written)}") - - -def main() -> None: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - - take = sub.add_parser("ingest", help="append one release's results to the history") - take.add_argument("--history", required=True) - take.add_argument("--results", required=True, help="directory holding BenchmarkDotNet output") - take.add_argument("--version", required=True) - take.add_argument("--commit", default="") - take.add_argument("--date", default="") - take.add_argument("--run-id", default="") - take.add_argument( - "--baseline-ns", - type=float, - default=None, - help="reference time to record when this version predates BaselineBenchmarks", - ) - take.set_defaults(func=ingest) - - draw = sub.add_parser("render", help="draw the history as SVG") - draw.add_argument("--history", required=True) - draw.add_argument("--out", required=True, help="light-mode path; the dark file sits beside it") - draw.set_defaults(func=render) - - args = parser.parse_args() - args.func(args) - - -if __name__ == "__main__": - main() From 301ed54a187875c41ce2cae510c5121b611d7470 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 03:36:41 +0000 Subject: [PATCH 3/4] Map the reports to parsed documents rather than reparsing in the body [patch] ReadReports opened each report inside the loop, which is a map written as a statement. Projecting with Select says the same thing in the loop header. Output is unchanged: re-ingesting the same reports and re-rendering leaves the history and both SVGs byte for byte identical. Reported by github-code-quality on #83. Its suggested `using System.Linq;` is not needed here, because implicit usings already cover it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017jrnV7N94UGL8fDRRE8Xt8 --- scripts/benchmark-history.cs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/scripts/benchmark-history.cs b/scripts/benchmark-history.cs index c96c609..732c293 100644 --- a/scripts/benchmark-history.cs +++ b/scripts/benchmark-history.cs @@ -206,9 +206,8 @@ private static (SortedDictionary> Measured, string C string cpu = ""; string runtime = ""; - foreach (string report in reports) + foreach (JsonNode document in reports.Select(report => JsonNode.Parse(File.ReadAllText(report))!)) { - JsonNode document = JsonNode.Parse(File.ReadAllText(report))!; if (cpu.Length == 0 && document["HostEnvironmentInfo"] is JsonNode environment) { cpu = (environment["ProcessorName"]?.GetValue() ?? "").Trim(); From 4d2d60d3f13080ff950176f2382fc7c89c3d029c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 04:21:37 +0000 Subject: [PATCH 4/4] Seed the chart with every release that can be measured the same way [patch] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight points now, v1.8.0 through v2.0.5, where there were six. Two were missing rather than unavailable: v1.8.0 is a real tag that also carries the benchmark project, and v2.0.5 has been released since the first seed. All eight are re-measured in one pass against one reading of the reference workload, so every point is comparable to every other by construction rather than by argument. v1.8.0 is where this can reach, and the boundary is not arbitrary: the benchmark project arrived in #68, so no earlier tag has one to run. Going further would mean running today's benchmarks against old released packages instead of each tag's own source — which works, and was tried: only ConstructionBenchmarks fails to compile against 1.7.36, because CreateFromComponents is internal and the InternalsVisibleTo that reaches it arrived with the suite. That is a different measurement for six of the seven benchmarks, so it is not mixed in here silently. The eighth point earns its place immediately. Divide allocated 312 bytes at v1.8.0 and 240 at v1.9.0, so the exact-division work in #69 shows as its own step before the value type takes another 40 off every operation at 2.0.0. The six-point seed started after that and showed none of it. The axis caption lists every release again. It thinned to every other label above six, which reads as the whole list and would claim there were fewer releases than there are; twelve is the point where they stop fitting. Also drops the two .gitignore entries the first commit added. ktsu.Sdk regenerates that file on every build and strips them, so committing them leaves a dirty tree after any build, and the workflow no longer writes to either path. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017jrnV7N94UGL8fDRRE8Xt8 --- .gitignore | 4 - docs/benchmarks/history.json | 454 +++++++++++++++++++-------- docs/benchmarks/performance-dark.svg | 226 +++++++------ docs/benchmarks/performance.svg | 226 +++++++------ scripts/benchmark-history.cs | 4 +- 5 files changed, 585 insertions(+), 329 deletions(-) diff --git a/.gitignore b/.gitignore index e7ff079..dc0470a 100644 --- a/.gitignore +++ b/.gitignore @@ -58,10 +58,6 @@ dlldata.c # Benchmark Results BenchmarkDotNet.Artifacts/ -# Scratch directories the Benchmark History workflow writes its runs into. -bench/ -backfill/ - # .NET Core project.lock.json project.fragment.lock.json diff --git a/docs/benchmarks/history.json b/docs/benchmarks/history.json index a7f8fc2..2d37639 100644 --- a/docs/benchmarks/history.json +++ b/docs/benchmarks/history.json @@ -1,102 +1,203 @@ { "schemaVersion": 1, "entries": [ + { + "version": "1.8.0", + "commit": "b0564d7", + "date": "2026-09-13", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 454.8992, + "runId": "local-seed", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 123.1859, + "allocatedBytes": 104 + }, + "Digits=30": { + "meanNs": 164.7113, + "allocatedBytes": 120 + }, + "Digits=200": { + "meanNs": 402.6075, + "allocatedBytes": 264 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 794.4781, + "allocatedBytes": 264 + }, + "Digits=30": { + "meanNs": 889.3647, + "allocatedBytes": 312 + }, + "Digits=200": { + "meanNs": 995.529, + "allocatedBytes": 456 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 81.9846, + "allocatedBytes": 72 + }, + "Digits=30": { + "meanNs": 184.0505, + "allocatedBytes": 96 + }, + "Digits=200": { + "meanNs": 1050.0351, + "allocatedBytes": 232 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 4.5, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 4.7745, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 5.1859, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 35.0957, + "allocatedBytes": 40 + }, + "Digits=30": { + "meanNs": 76.4736, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 254.6939, + "allocatedBytes": 40 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 20.149, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 199.5932, + "allocatedBytes": 40 + }, + "Digits=30": { + "meanNs": 480.1194, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 2534.6838, + "allocatedBytes": 152 + } + } + } + }, { "version": "1.9.0", "commit": "cd9a822", "date": "2026-09-13", "cpu": "Intel Xeon Processor 2.80GHz", "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", - "baselineNs": 456.3006, - "runId": "local-backfill", + "baselineNs": 454.8992, + "runId": "local-seed", "benchmarks": { "ArithmeticBenchmarks.Add": { "Digits=8": { - "meanNs": 118.4378, + "meanNs": 114.0296, "allocatedBytes": 104 }, "Digits=30": { - "meanNs": 157.0446, + "meanNs": 156.3233, "allocatedBytes": 120 }, "Digits=200": { - "meanNs": 401.5004, + "meanNs": 428.4019, "allocatedBytes": 264 } }, "ArithmeticBenchmarks.Divide": { "Digits=8": { - "meanNs": 408.9082, + "meanNs": 403.9122, "allocatedBytes": 192 }, "Digits=30": { - "meanNs": 597.664, + "meanNs": 613.6075, "allocatedBytes": 240 }, "Digits=200": { - "meanNs": 3397.0807, + "meanNs": 2375.4439, "allocatedBytes": 568 } }, "ArithmeticBenchmarks.Multiply": { "Digits=8": { - "meanNs": 85.9557, + "meanNs": 84.6192, "allocatedBytes": 72 }, "Digits=30": { - "meanNs": 180.616, + "meanNs": 179.2593, "allocatedBytes": 96 }, "Digits=200": { - "meanNs": 1017.6489, + "meanNs": 1029.5614, "allocatedBytes": 232 } }, "ComparisonBenchmarks.CompareTo": { "Digits=8": { - "meanNs": 4.9821, + "meanNs": 4.55, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 4.7751, + "meanNs": 4.7625, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 5.1426, + "meanNs": 4.5483, "allocatedBytes": 0 } }, "ConstructionBenchmarks.Sanitizing": { "Digits=8": { - "meanNs": 33.1727, + "meanNs": 35.4251, "allocatedBytes": 40 }, "Digits=30": { - "meanNs": 76.8865, + "meanNs": 80.1594, "allocatedBytes": 40 }, "Digits=200": { - "meanNs": 257.1197, + "meanNs": 262.4124, "allocatedBytes": 40 } }, "ConversionBenchmarks.ToDouble": { "": { - "meanNs": 20.6804, + "meanNs": 20.7065, "allocatedBytes": 0 } }, "TextBenchmarks.Parse": { "Digits=8": { - "meanNs": 205.5837, + "meanNs": 205.5582, "allocatedBytes": 40 }, "Digits=30": { - "meanNs": 489.1961, + "meanNs": 490.8118, "allocatedBytes": 80 }, "Digits=200": { - "meanNs": 2436.9512, + "meanNs": 2503.218, "allocatedBytes": 152 } } @@ -108,96 +209,96 @@ "date": "2026-09-13", "cpu": "Intel Xeon Processor 2.80GHz", "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", - "baselineNs": 456.3006, - "runId": "local-backfill", + "baselineNs": 454.8992, + "runId": "local-seed", "benchmarks": { "ArithmeticBenchmarks.Add": { "Digits=8": { - "meanNs": 102.2084, + "meanNs": 100.4854, "allocatedBytes": 64 }, "Digits=30": { - "meanNs": 144.8577, + "meanNs": 145.0197, "allocatedBytes": 80 }, "Digits=200": { - "meanNs": 401.2804, + "meanNs": 406.8626, "allocatedBytes": 224 } }, "ArithmeticBenchmarks.Divide": { "Digits=8": { - "meanNs": 425.5046, + "meanNs": 421.971, "allocatedBytes": 152 }, "Digits=30": { - "meanNs": 636.3521, + "meanNs": 649.0157, "allocatedBytes": 200 }, "Digits=200": { - "meanNs": 3408.2489, + "meanNs": 2501.2133, "allocatedBytes": 528 } }, "ArithmeticBenchmarks.Multiply": { "Digits=8": { - "meanNs": 66.8426, + "meanNs": 70.0239, "allocatedBytes": 32 }, "Digits=30": { - "meanNs": 167.6952, + "meanNs": 175.1806, "allocatedBytes": 56 }, "Digits=200": { - "meanNs": 1011.6339, + "meanNs": 1025.2062, "allocatedBytes": 192 } }, "ComparisonBenchmarks.CompareTo": { "Digits=8": { - "meanNs": 3.7839, + "meanNs": 3.1494, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 3.7399, + "meanNs": 4.2118, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 3.5946, + "meanNs": 3.589, "allocatedBytes": 0 } }, "ConstructionBenchmarks.Sanitizing": { "Digits=8": { - "meanNs": 19.1305, + "meanNs": 19.0883, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 64.8693, + "meanNs": 63.0727, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 245.6405, + "meanNs": 242.1056, "allocatedBytes": 0 } }, "ConversionBenchmarks.ToDouble": { "": { - "meanNs": 5.6483, + "meanNs": 5.749, "allocatedBytes": 0 } }, "TextBenchmarks.Parse": { "Digits=8": { - "meanNs": 185.9653, + "meanNs": 187.9601, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 478.1012, + "meanNs": 479.94, "allocatedBytes": 40 }, "Digits=200": { - "meanNs": 2545.4556, + "meanNs": 2528.7065, "allocatedBytes": 112 } } @@ -209,96 +310,96 @@ "date": "2026-09-14", "cpu": "Intel Xeon Processor 2.80GHz", "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", - "baselineNs": 456.3006, - "runId": "local-backfill", + "baselineNs": 454.8992, + "runId": "local-seed", "benchmarks": { "ArithmeticBenchmarks.Add": { "Digits=8": { - "meanNs": 98.0732, + "meanNs": 100.8056, "allocatedBytes": 64 }, "Digits=30": { - "meanNs": 143.3102, + "meanNs": 148.6598, "allocatedBytes": 80 }, "Digits=200": { - "meanNs": 388.1712, + "meanNs": 576.7779, "allocatedBytes": 224 } }, "ArithmeticBenchmarks.Divide": { "Digits=8": { - "meanNs": 421.3311, + "meanNs": 432.8415, "allocatedBytes": 152 }, "Digits=30": { - "meanNs": 639.3545, + "meanNs": 630.3228, "allocatedBytes": 200 }, "Digits=200": { - "meanNs": 2377.5382, + "meanNs": 2408.9761, "allocatedBytes": 528 } }, "ArithmeticBenchmarks.Multiply": { "Digits=8": { - "meanNs": 66.1012, + "meanNs": 71.7227, "allocatedBytes": 32 }, "Digits=30": { - "meanNs": 166.4438, + "meanNs": 168.2777, "allocatedBytes": 56 }, "Digits=200": { - "meanNs": 1005.4831, + "meanNs": 1036.3418, "allocatedBytes": 192 } }, "ComparisonBenchmarks.CompareTo": { "Digits=8": { - "meanNs": 3.6457, + "meanNs": 3.9038, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 3.9055, + "meanNs": 4.1531, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 3.8933, + "meanNs": 4.2655, "allocatedBytes": 0 } }, "ConstructionBenchmarks.Sanitizing": { "Digits=8": { - "meanNs": 19.1771, + "meanNs": 19.3621, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 63.2906, + "meanNs": 63.2541, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 241.3681, + "meanNs": 242.4946, "allocatedBytes": 0 } }, "ConversionBenchmarks.ToDouble": { "": { - "meanNs": 5.7523, + "meanNs": 5.0037, "allocatedBytes": 0 } }, "TextBenchmarks.Parse": { "Digits=8": { - "meanNs": 187.7729, + "meanNs": 186.071, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 485.3833, + "meanNs": 486.0585, "allocatedBytes": 40 }, "Digits=200": { - "meanNs": 2510.739, + "meanNs": 2492.1256, "allocatedBytes": 112 } } @@ -310,96 +411,96 @@ "date": "2026-09-14", "cpu": "Intel Xeon Processor 2.80GHz", "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", - "baselineNs": 456.3006, - "runId": "local-backfill", + "baselineNs": 454.8992, + "runId": "local-seed", "benchmarks": { "ArithmeticBenchmarks.Add": { "Digits=8": { - "meanNs": 106.141, + "meanNs": 101.1571, "allocatedBytes": 64 }, "Digits=30": { - "meanNs": 151.7249, + "meanNs": 147.8035, "allocatedBytes": 80 }, "Digits=200": { - "meanNs": 412.2945, + "meanNs": 397.9905, "allocatedBytes": 224 } }, "ArithmeticBenchmarks.Divide": { "Digits=8": { - "meanNs": 428.2106, + "meanNs": 421.5628, "allocatedBytes": 152 }, "Digits=30": { - "meanNs": 647.3638, + "meanNs": 648.9157, "allocatedBytes": 200 }, "Digits=200": { - "meanNs": 2435.1219, + "meanNs": 2435.5906, "allocatedBytes": 528 } }, "ArithmeticBenchmarks.Multiply": { "Digits=8": { - "meanNs": 72.0185, + "meanNs": 74.7106, "allocatedBytes": 32 }, "Digits=30": { - "meanNs": 172.6775, + "meanNs": 169.6895, "allocatedBytes": 56 }, "Digits=200": { - "meanNs": 1021.6319, + "meanNs": 1011.6622, "allocatedBytes": 192 } }, "ComparisonBenchmarks.CompareTo": { "Digits=8": { - "meanNs": 4.1897, + "meanNs": 4.0939, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 3.6436, + "meanNs": 3.8344, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 4.4223, + "meanNs": 3.6637, "allocatedBytes": 0 } }, "ConstructionBenchmarks.Sanitizing": { "Digits=8": { - "meanNs": 19.5526, + "meanNs": 20.3191, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 64.9609, + "meanNs": 64.0998, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 243.2625, + "meanNs": 244.0778, "allocatedBytes": 0 } }, "ConversionBenchmarks.ToDouble": { "": { - "meanNs": 4.7889, + "meanNs": 5.3519, "allocatedBytes": 0 } }, "TextBenchmarks.Parse": { "Digits=8": { - "meanNs": 188.7315, + "meanNs": 185.887, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 502.7654, + "meanNs": 489.6898, "allocatedBytes": 40 }, "Digits=200": { - "meanNs": 2469.2835, + "meanNs": 2449.7325, "allocatedBytes": 112 } } @@ -411,96 +512,96 @@ "date": "2026-09-14", "cpu": "Intel Xeon Processor 2.80GHz", "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", - "baselineNs": 456.3006, - "runId": "local-backfill", + "baselineNs": 454.8992, + "runId": "local-seed", "benchmarks": { "ArithmeticBenchmarks.Add": { "Digits=8": { - "meanNs": 99.4048, + "meanNs": 106.0063, "allocatedBytes": 64 }, "Digits=30": { - "meanNs": 149.9368, + "meanNs": 145.0424, "allocatedBytes": 80 }, "Digits=200": { - "meanNs": 417.451, + "meanNs": 400.2685, "allocatedBytes": 224 } }, "ArithmeticBenchmarks.Divide": { "Digits=8": { - "meanNs": 436.516, + "meanNs": 423.013, "allocatedBytes": 152 }, "Digits=30": { - "meanNs": 641.4446, + "meanNs": 641.3537, "allocatedBytes": 200 }, "Digits=200": { - "meanNs": 3476.5374, + "meanNs": 3414.9285, "allocatedBytes": 528 } }, "ArithmeticBenchmarks.Multiply": { "Digits=8": { - "meanNs": 72.2699, + "meanNs": 67.2843, "allocatedBytes": 32 }, "Digits=30": { - "meanNs": 171.918, + "meanNs": 168.3793, "allocatedBytes": 56 }, "Digits=200": { - "meanNs": 1027.3061, + "meanNs": 1019.9757, "allocatedBytes": 192 } }, "ComparisonBenchmarks.CompareTo": { "Digits=8": { - "meanNs": 4.2138, + "meanNs": 4.4645, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 3.828, + "meanNs": 3.8372, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 3.5863, + "meanNs": 3.5919, "allocatedBytes": 0 } }, "ConstructionBenchmarks.Sanitizing": { "Digits=8": { - "meanNs": 20.0353, + "meanNs": 19.6194, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 62.9153, + "meanNs": 67.0253, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 241.841, + "meanNs": 247.2198, "allocatedBytes": 0 } }, "ConversionBenchmarks.ToDouble": { "": { - "meanNs": 5.3217, + "meanNs": 5.1391, "allocatedBytes": 0 } }, "TextBenchmarks.Parse": { "Digits=8": { - "meanNs": 192.7158, + "meanNs": 190.2204, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 472.4866, + "meanNs": 485.4024, "allocatedBytes": 40 }, "Digits=200": { - "meanNs": 2528.1408, + "meanNs": 2529.4197, "allocatedBytes": 112 } } @@ -512,96 +613,197 @@ "date": "2026-09-15", "cpu": "Intel Xeon Processor 2.80GHz", "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", - "baselineNs": 456.3006, - "runId": "local-backfill", + "baselineNs": 454.8992, + "runId": "local-seed", + "benchmarks": { + "ArithmeticBenchmarks.Add": { + "Digits=8": { + "meanNs": 103.5012, + "allocatedBytes": 64 + }, + "Digits=30": { + "meanNs": 147.654, + "allocatedBytes": 80 + }, + "Digits=200": { + "meanNs": 395.7874, + "allocatedBytes": 224 + } + }, + "ArithmeticBenchmarks.Divide": { + "Digits=8": { + "meanNs": 419.0343, + "allocatedBytes": 152 + }, + "Digits=30": { + "meanNs": 627.7278, + "allocatedBytes": 200 + }, + "Digits=200": { + "meanNs": 2440.4735, + "allocatedBytes": 528 + } + }, + "ArithmeticBenchmarks.Multiply": { + "Digits=8": { + "meanNs": 69.7283, + "allocatedBytes": 32 + }, + "Digits=30": { + "meanNs": 165.934, + "allocatedBytes": 56 + }, + "Digits=200": { + "meanNs": 1027.7703, + "allocatedBytes": 192 + } + }, + "ComparisonBenchmarks.CompareTo": { + "Digits=8": { + "meanNs": 3.5588, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 3.5576, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 3.6354, + "allocatedBytes": 0 + } + }, + "ConstructionBenchmarks.Sanitizing": { + "Digits=8": { + "meanNs": 19.7243, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 63.6485, + "allocatedBytes": 0 + }, + "Digits=200": { + "meanNs": 241.9847, + "allocatedBytes": 0 + } + }, + "ConversionBenchmarks.ToDouble": { + "": { + "meanNs": 5.3881, + "allocatedBytes": 0 + } + }, + "TextBenchmarks.Parse": { + "Digits=8": { + "meanNs": 189.778, + "allocatedBytes": 0 + }, + "Digits=30": { + "meanNs": 477.5123, + "allocatedBytes": 40 + }, + "Digits=200": { + "meanNs": 2512.168, + "allocatedBytes": 112 + } + } + } + }, + { + "version": "2.0.5", + "commit": "3857bac", + "date": "2026-09-16", + "cpu": "Intel Xeon Processor 2.80GHz", + "runtime": ".NET 10.0.12 (10.0.12, 10.0.1226.42308)", + "baselineNs": 454.8992, + "runId": "local-seed", "benchmarks": { "ArithmeticBenchmarks.Add": { "Digits=8": { - "meanNs": 102.0636, + "meanNs": 104.0341, "allocatedBytes": 64 }, "Digits=30": { - "meanNs": 149.3164, + "meanNs": 144.1285, "allocatedBytes": 80 }, "Digits=200": { - "meanNs": 409.0072, + "meanNs": 397.1837, "allocatedBytes": 224 } }, "ArithmeticBenchmarks.Divide": { "Digits=8": { - "meanNs": 425.7428, + "meanNs": 425.7019, "allocatedBytes": 152 }, "Digits=30": { - "meanNs": 642.9308, + "meanNs": 615.669, "allocatedBytes": 200 }, "Digits=200": { - "meanNs": 3448.2813, + "meanNs": 3372.9288, "allocatedBytes": 528 } }, "ArithmeticBenchmarks.Multiply": { "Digits=8": { - "meanNs": 69.2106, + "meanNs": 68.7651, "allocatedBytes": 32 }, "Digits=30": { - "meanNs": 173.0146, + "meanNs": 167.7462, "allocatedBytes": 56 }, "Digits=200": { - "meanNs": 1028.6434, + "meanNs": 1010.8013, "allocatedBytes": 192 } }, "ComparisonBenchmarks.CompareTo": { "Digits=8": { - "meanNs": 4.1475, + "meanNs": 4.2163, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 3.9174, + "meanNs": 3.8202, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 4.1474, + "meanNs": 3.5939, "allocatedBytes": 0 } }, "ConstructionBenchmarks.Sanitizing": { "Digits=8": { - "meanNs": 19.5191, + "meanNs": 19.1214, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 63.7933, + "meanNs": 62.9138, "allocatedBytes": 0 }, "Digits=200": { - "meanNs": 243.2835, + "meanNs": 241.3521, "allocatedBytes": 0 } }, "ConversionBenchmarks.ToDouble": { "": { - "meanNs": 4.7245, + "meanNs": 4.8894, "allocatedBytes": 0 } }, "TextBenchmarks.Parse": { "Digits=8": { - "meanNs": 185.7342, + "meanNs": 194.2737, "allocatedBytes": 0 }, "Digits=30": { - "meanNs": 484.1852, + "meanNs": 476.8454, "allocatedBytes": 40 }, "Digits=200": { - "meanNs": 2339.7804, + "meanNs": 2442.8716, "allocatedBytes": 112 } } diff --git a/docs/benchmarks/performance-dark.svg b/docs/benchmarks/performance-dark.svg index 1f683f6..28fc8de 100644 --- a/docs/benchmarks/performance-dark.svg +++ b/docs/benchmarks/performance-dark.svg @@ -11,84 +11,98 @@ PreciseNumber performance by release -6 releases · newest 2.0.4 · 2026-09-15 +8 releases · newest 2.0.5 · 2026-09-16 Allocated bytes per operation Deterministic: the same code allocates the same bytes on any machine. Add (30 digits) - + - - - - + + + + + + 80 B 120 B Multiply (30 digits) - + - - - - + + + + + + 56 B 96 B Divide (30 digits) - + - - - - - -200 B -240 B + + + + + + + +200 B +312 B CompareTo (30 digits) - + - - - - + + + + + + 0 B 0 B Construct (30 digits) - + - - - - + + + + + + 0 B 40 B Parse (30 digits) - + - - - - + + + + + + 40 B 80 B ToDouble - + - - - - + + + + + + 0 B 0 B @@ -97,81 +111,95 @@ Divided by a reference loop measured in the same job, which cancels most of the difference between CI runners. Lower is faster. Add (30 digits) - + - - - - - -0.327× -0.344× + + + + + + + +0.317× +0.362× Multiply (30 digits) - + - - - - - -0.379× -0.396× + + + + + + + +0.369× +0.405× Divide (30 digits) - - - - - - - -1.41× -1.31× + + + + + + + + + +1.35× +1.96× CompareTo (30 digits) - + - - - - - -0.0086× + + + + + + + +0.0084× 0.0105× Construct (30 digits) - - - - - - - -0.140× -0.168× + + + + + + + + + +0.138× +0.168× Parse (30 digits) - - - - - - - -1.06× -1.07× + + + + + + + + + +1.05× +1.06× ToDouble - - - - - - - -0.0104× -0.0453× -releases, oldest to newest: 1.9.0 → 2.0.0 → 2.0.1 → 2.0.2 → 2.0.3 → 2.0.4 + + + + + + + + + +0.0107× +0.0443× +releases, oldest to newest: 1.8.0 → 1.9.0 → 2.0.0 → 2.0.1 → 2.0.2 → 2.0.3 → 2.0.4 → 2.0.5 Measured on Intel Xeon Processor 2.80GHz. Full tables: PreciseNumber.Benchmarks. diff --git a/docs/benchmarks/performance.svg b/docs/benchmarks/performance.svg index 8e6fac5..b417420 100644 --- a/docs/benchmarks/performance.svg +++ b/docs/benchmarks/performance.svg @@ -11,84 +11,98 @@ PreciseNumber performance by release -6 releases · newest 2.0.4 · 2026-09-15 +8 releases · newest 2.0.5 · 2026-09-16 Allocated bytes per operation Deterministic: the same code allocates the same bytes on any machine. Add (30 digits) - + - - - - + + + + + + 80 B 120 B Multiply (30 digits) - + - - - - + + + + + + 56 B 96 B Divide (30 digits) - + - - - - - -200 B -240 B + + + + + + + +200 B +312 B CompareTo (30 digits) - + - - - - + + + + + + 0 B 0 B Construct (30 digits) - + - - - - + + + + + + 0 B 40 B Parse (30 digits) - + - - - - + + + + + + 40 B 80 B ToDouble - + - - - - + + + + + + 0 B 0 B @@ -97,81 +111,95 @@ Divided by a reference loop measured in the same job, which cancels most of the difference between CI runners. Lower is faster. Add (30 digits) - + - - - - - -0.327× -0.344× + + + + + + + +0.317× +0.362× Multiply (30 digits) - + - - - - - -0.379× -0.396× + + + + + + + +0.369× +0.405× Divide (30 digits) - - - - - - - -1.41× -1.31× + + + + + + + + + +1.35× +1.96× CompareTo (30 digits) - + - - - - - -0.0086× + + + + + + + +0.0084× 0.0105× Construct (30 digits) - - - - - - - -0.140× -0.168× + + + + + + + + + +0.138× +0.168× Parse (30 digits) - - - - - - - -1.06× -1.07× + + + + + + + + + +1.05× +1.06× ToDouble - - - - - - - -0.0104× -0.0453× -releases, oldest to newest: 1.9.0 → 2.0.0 → 2.0.1 → 2.0.2 → 2.0.3 → 2.0.4 + + + + + + + + + +0.0107× +0.0443× +releases, oldest to newest: 1.8.0 → 1.9.0 → 2.0.0 → 2.0.1 → 2.0.2 → 2.0.3 → 2.0.4 → 2.0.5 Measured on Intel Xeon Processor 2.80GHz. Full tables: PreciseNumber.Benchmarks. diff --git a/scripts/benchmark-history.cs b/scripts/benchmark-history.cs index 732c293..408895d 100644 --- a/scripts/benchmark-history.cs +++ b/scripts/benchmark-history.cs @@ -489,7 +489,9 @@ private static void Footer(StringBuilder svg, JsonArray entries, string[] labels List ticks = []; for (int index = 0; index < labels.Length; index++) { - if (labels.Length > 6 && index > 0 && index < labels.Length - 1 && index % 2 == 1) + // Every label while they fit. Thinning them reads as the whole list, which would say + // there were fewer releases than there were. + if (labels.Length > 12 && index > 0 && index < labels.Length - 1 && index % 2 == 1) { continue; }