Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 204 additions & 0 deletions .github/workflows/benchmark-history.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
name: Benchmark History

# Measures a small, fixed set of benchmarks once per release, appends the numbers to a committed
# history file, and redraws the chart the README shows. Deliberately separate from benchmarks.yml,
# which exists to get a full ad-hoc run on demand and publishes nothing.
#
# Two ways in:
# * a published release, which measures that version and adds one point;
# * a manual dispatch listing refs, which measures each of them in ONE job and backfills.
#
# The backfill running as a single job is the point rather than an optimisation. Separate runs land
# on different CI hosts, and the difference between an x86-64-v3 and a v4 runner is larger than
# most releases are, so points gathered in separate jobs are not comparable as raw times. Within
# one job they are. Across jobs, BaselineBenchmarks is what ties them together -- see its remarks.

on:
release:
types: [published]
workflow_dispatch:
inputs:
refs:
description: "Space-separated refs to backfill, oldest first (tags, branches, or SHAs)"
required: false
default: "cd9a8227793bbd6ea791be7f6c0579ae1242eec5 v2.0.0 v2.0.1 v2.0.2 v2.0.3 v2.0.4"
type: string
labels:
description: "Optional space-separated version labels matching refs, when a ref is not a version"
required: false
default: "1.9.0 2.0.0 2.0.1 2.0.2 2.0.3 2.0.4"
type: string

permissions:
contents: write

concurrency:
group: benchmark-history
cancel-in-progress: false

env:
DOTNET_VERSION: "10.0"
HISTORY: docs/benchmarks/history.json
# Already ignored, and ktsu.Sdk regenerates .gitignore on build so a new entry would not last.
RUNS: BenchmarkDotNet.Artifacts
CHART: docs/benchmarks/performance.svg
# The set drawn in the README. Name globs rather than a [BenchmarkCategory], because the backfill
# runs this same filter against older checkouts that predate any attribute added today.
HEADLINE_FILTER: >-
*ArithmeticBenchmarks.Add
*ArithmeticBenchmarks.Multiply
*ArithmeticBenchmarks.Divide
*ComparisonBenchmarks.CompareTo
*ConstructionBenchmarks.Sanitizing
*TextBenchmarks.Parse
*ConversionBenchmarks.ToDouble
# Short runs: three iterations is enough for a trend line, and a release should not tie up a
# runner for half an hour. benchmarks.yml is still there for a full-length run.
BENCHMARK_JOB: short

jobs:
measure:
name: Measure and chart
runs-on: ubuntu-latest
timeout-minutes: 180

steps:
- name: Checkout Repository
uses: actions/checkout@v7
with:
# The default branch, not the released tag. The results are committed back here, and a
# release event would otherwise leave the checkout detached at the tag, so the push at
# the end would be asking the default branch to move backwards. The tag itself is
# measured through a worktree below, exactly as a backfill ref is.
ref: ${{ github.event.repository.default_branch }}
fetch-depth: 0

- name: Setup .NET SDK ${{ env.DOTNET_VERSION }}
uses: actions/setup-dotnet@v6
with:
dotnet-version: ${{ env.DOTNET_VERSION }}.x

# Measured from this checkout, and stamped onto every entry this job produces. Older refs do
# not carry BaselineBenchmarks, and do not need to: everything measured in this job shares
# one runner, so one reading of that runner describes all of them.
- name: Measure the reference workload
id: baseline
shell: bash
run: |
set -euo pipefail
dotnet run -c Release --project PreciseNumber.Benchmarks -- \
--filter '*BaselineBenchmarks.ReferenceWork' \
--job "$BENCHMARK_JOB" \
--artifacts "$GITHUB_WORKSPACE/$RUNS/baseline"
ns=$(dotnet run scripts/benchmark-history.cs -- baseline --results "$RUNS/baseline")
echo "Reference workload: $ns ns"
echo "ns=$ns" >> "$GITHUB_OUTPUT"

- name: Measure the released version
if: github.event_name == 'release'
shell: bash
env:
VERSION: ${{ github.event.release.tag_name }}
run: |
set -euo pipefail
version="${VERSION#v}"
work="${RUNNER_TEMP}/bench-$version"
git worktree add --detach "$work" "$VERSION"

(cd "$work" && dotnet run -c Release --project PreciseNumber.Benchmarks -- \
--filter $HEADLINE_FILTER \
--job "$BENCHMARK_JOB" \
--artifacts "$GITHUB_WORKSPACE/$RUNS/$version")

dotnet run scripts/benchmark-history.cs -- ingest \
--history "$HISTORY" \
--results "$RUNS/$version" \
--version "$version" \
--commit "$(git rev-parse --short "$VERSION^{commit}")" \
--date "$(git log -1 --format=%cs "$VERSION")" \
--run-id "${{ github.run_id }}" \
--baseline-ns "${{ steps.baseline.outputs.ns }}"

git worktree remove --force "$work"

- name: Measure each backfill ref
if: github.event_name == 'workflow_dispatch'
shell: bash
env:
REFS: ${{ inputs.refs }}
LABELS: ${{ inputs.labels }}
BASELINE_NS: ${{ steps.baseline.outputs.ns }}
run: |
set -euo pipefail
read -ra refs <<< "$REFS"
read -ra labels <<< "$LABELS"

for index in "${!refs[@]}"; do
ref="${refs[$index]}"
label="${labels[$index]:-${ref#v}}"
work="${RUNNER_TEMP}/bench-$label"

echo "::group::$label ($ref)"
rm -rf "$work"
git worktree add --detach "$work" "$ref"

if [ ! -f "$work/PreciseNumber.Benchmarks/PreciseNumber.Benchmarks.csproj" ]; then
echo "::warning::$ref has no benchmark project; skipping"
git worktree remove --force "$work"
echo "::endgroup::"
continue
fi

# Each ref is measured by its own benchmark sources. Between 2.0.0 and now those
# sources are unchanged, so this compares library versions rather than harnesses.
(cd "$work" && dotnet run -c Release --project PreciseNumber.Benchmarks -- \
--filter $HEADLINE_FILTER \
--job "$BENCHMARK_JOB" \
--artifacts "$GITHUB_WORKSPACE/$RUNS/$label")

dotnet run scripts/benchmark-history.cs -- ingest \
--history "$HISTORY" \
--results "$RUNS/$label" \
--version "$label" \
--commit "$(git rev-parse --short "$ref^{commit}")" \
--date "$(git log -1 --format=%cs "$ref")" \
--run-id "${{ github.run_id }}" \
--baseline-ns "$BASELINE_NS"

git worktree remove --force "$work"
echo "::endgroup::"
done

- name: Redraw the chart
shell: bash
run: dotnet run scripts/benchmark-history.cs -- render --history "$HISTORY" --out "$CHART"

- name: Commit the history and the chart
shell: bash
run: |
set -euo pipefail
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
# Staged first, then compared against the index: on the first run these files are new,
# and `git diff` alone does not see an untracked file, so the run would push nothing and
# still report success.
git add "$HISTORY" "${CHART%.svg}"*.svg
if git diff --cached --quiet; then
echo "Nothing changed."
exit 0
fi
# [skip ci] so that committing results does not start the pipeline over again.
git commit -m "[bot][skip ci] Update benchmark history"
branch="${{ github.event.repository.default_branch }}"
# Another release may have landed while this job was measuring.
git pull --rebase origin "$branch"
git push origin "HEAD:$branch"

- name: Upload the raw reports
if: always()
uses: actions/upload-artifact@v7
with:
name: benchmark-history-${{ github.run_id }}
path: ${{ env.RUNS }}/
retention-days: 30
if-no-files-found: warn
59 changes: 59 additions & 0 deletions PreciseNumber.Benchmarks/BaselineBenchmarks.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// Copyright (c) 2023-2026 ktsu-dev contributors

namespace ktsu.PreciseNumber.Benchmarks;

using BenchmarkDotNet.Attributes;

/// <summary>
/// Measures a fixed workload that touches none of this library, so that timings taken on
/// different machines can be compared.
/// </summary>
/// <remarks>
/// Every release is benchmarked in its own CI job, and a job lands on whichever shared runner is
/// free — an x86-64-v3 or v4 host, at whatever clock its neighbours leave it. That difference is
/// routinely larger than the changes a release makes, so a chart of raw times across releases
/// mostly plots the runner.
/// <para>
/// This benchmark is the fixed point that makes the rest comparable. It is integer arithmetic over
/// a value the JIT cannot fold away, chosen because it has no allocation, no library code, and no
/// dependence on anything that changes between versions — so its measured time is a reading of the
/// machine and nothing else. Dividing a benchmark's time by this one's, taken in the same job,
/// cancels most of the difference between hosts. `scripts/benchmark_history.py` records it on
/// every entry and plots the ratio rather than the nanoseconds.
/// </para>
/// <para>
/// It follows that this method's body must never change. Editing it silently rescales every
/// comparison drawn against history recorded before the edit.
/// </para>
/// </remarks>
[MemoryDiagnoser]
public class BaselineBenchmarks
{
// Read from a field rather than written as a literal, so that the loop cannot be constant
// folded into its own answer at JIT time.
private ulong seed;

/// <summary>
/// Sets the starting value.
/// </summary>
[GlobalSetup]
public void Setup() => seed = 0xcbf29ce484222325;

/// <summary>
/// Mixes a counter with a multiply-xor-shift step, the way a non-cryptographic hash does.
/// </summary>
/// <returns>The accumulated value, returned so that nothing here is dead code.</returns>
[Benchmark]
public ulong ReferenceWork()
{
ulong accumulator = seed;

for (int i = 0; i < 256; i++)
{
accumulator = (accumulator ^ (ulong)i) * 0x100000001b3;
accumulator ^= accumulator >> 29;
}

return accumulator;
}
}
59 changes: 37 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ A high-precision numeric type for .NET that provides arbitrary precision arithme

- [Features](#features)

- [Performance](#performance)

- [Getting Started](#getting-started)

- [Installation](#installation)
Expand Down Expand Up @@ -56,8 +58,6 @@ A high-precision numeric type for .NET that provides arbitrary precision arithme

- [Limitations](#limitations)

- [Performance](#performance)

- [API Reference](#api-reference)

- [PreciseNumber Class](#precisenumber-class)
Expand Down Expand Up @@ -86,6 +86,41 @@ A high-precision numeric type for .NET that provides arbitrary precision arithme

- **Balanced Performance**: The design prioritizes accuracy and precision while maintaining reasonable performance. For calculations where extreme precision matters more than raw speed, PreciseNumber delivers excellent results, though built-in numeric types remain faster for standard precision needs.

## Performance

Values are immutable value types. Every operation returns a new value, but that value lives inline
in its variable, field, or array element, so the only heap allocation is the `BigInteger` digit
array, and a significand that fits in an `int` doesn't need one. Cost therefore tracks the number
of significant digits rather than the magnitude of the value, and allocation matters as much as
raw speed.

<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/benchmarks/performance-dark.svg">
<img alt="Allocated bytes per operation, and time relative to a fixed reference workload, for each PreciseNumber release" src="docs/benchmarks/performance.svg">
</picture>

Every release measures a fixed set of benchmarks and adds a point to the chart above; the numbers
behind it are in [`docs/benchmarks/history.json`](docs/benchmarks/history.json).

Read the two halves differently. **Allocation is exact** — the same code allocates the same bytes on
any machine, so a step in the top row is always a real change. **Time is measured on shared CI
runners**, where the host a job happens to land on varies more than most releases do, so each time
is divided by a reference workload measured in the same job. That cancels most of the difference
between machines; what is left is indicative rather than precise, and a small wobble between two
releases is more likely the runner than the library.

The repository carries a [BenchmarkDotNet suite](PreciseNumber.Benchmarks/README.md) covering
construction, comparison, arithmetic, rounding, text conversion and primitive conversion, each
parameterised across 8, 30 and 200 significant digits:

```bash
dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*ArithmeticBenchmarks*'
```

Run it before and after any change to the library's internals. A full run can also be started
from the **Benchmarks** workflow in GitHub Actions, which archives the reports against the commit
that produced them.

# Getting Started

## Installation
Expand Down Expand Up @@ -394,26 +429,6 @@ three-argument overload when you want something other than that.

- Converting from `double`, `float`, or `Half` keeps the shortest digits that round-trip, so converting back gives the original value, and `0.3048` stays exactly 0.3048. The result of `0.1 + 0.2` in `double` arrives as 0.30000000000000004, because that's the value the `double` holds

## Performance

Values are immutable value types. Every operation returns a new value, but that value lives inline
in its variable, field, or array element, so the only heap allocation is the `BigInteger` digit
array, and a significand that fits in an `int` doesn't need one. Cost therefore tracks the number
of significant digits rather than the magnitude of the value, and allocation matters as much as
raw speed.

The repository carries a [BenchmarkDotNet suite](PreciseNumber.Benchmarks/README.md) covering
construction, comparison, arithmetic, rounding, text conversion and primitive conversion, each
parameterised across 8, 30 and 200 significant digits:

```bash
dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*ArithmeticBenchmarks*'
```

Run it before and after any change to the library's internals. A full run can also be started
from the **Benchmarks** workflow in GitHub Actions, which archives the reports against the commit
that produced them.

## API Reference

### PreciseNumber Class
Expand Down
Loading