Problem
The only performance gate benchmarks offer is @max_ms, an absolute per-function threshold parsed at src/benchmark/annotations.sh:29. Absolute thresholds are the wrong tool for CI:
- They have to be set loose enough to survive the slowest runner, so they only catch catastrophic regressions.
- They say nothing about a 30% slowdown that stays under the ceiling — which is exactly the regression worth catching.
- They need hand-tuning on every machine.
bashunit itself has run several performance campaigns (#761, #798, #830, #977-#979) where the actual question was always "is this slower than before", and answering it required manual before/after comparison.
Depends on #1028 — the --report-json file it adds is the baseline format.
Proposal
--baseline <file> Compare this run against a previous --report-json result
--baseline-tolerance <pct> Allowed regression before failing (default: 10)
--baseline-update <file> Write the current run as the new baseline
- Each benchmark is compared against its entry in the baseline by name.
- Exit non-zero when any benchmark is slower than
baseline * (1 + tolerance/100). - Output shows a per-benchmark delta (
+12.4%, -3.1%) so improvements are visible too. - Benchmarks missing from the baseline are reported as new and do not fail the run.
- Benchmarks in the baseline but absent from the run are reported as removed.
- Comparison uses the median, not the mean — document the choice.
Acceptance criteria
Repo checklist (agent)
- TDD: RED → GREEN → REFACTOR. Write the failing test first.
- Bash 3.0+ only: no
printf -v, no += append, no declare -A, no [[ ]], no ${var,,}, no &>>, no ${arr[-1]}. Expanding a possibly-empty array under set -u needs ${arr[@]+"${arr[@]}"}. - A new CLI flag must be wired in all of these or a parity test fails:
- Gates:
make sa, make lint, ./bashunit tests/, ./bashunit --parallel tests/. Never run shfmt -w. - Docs: update
docs/command-line.md. Editing docs/assertions.md invalidates the bashunit doc acceptance snapshot — regenerate it. - CHANGELOG.md: add one line under
## Unreleased. - Fixtures under
tests/acceptance/fixtures/ must not end in *test.sh. - One issue = one PR.
Problem
The only performance gate benchmarks offer is
@max_ms, an absolute per-function threshold parsed atsrc/benchmark/annotations.sh:29. Absolute thresholds are the wrong tool for CI:bashunit itself has run several performance campaigns (#761, #798, #830, #977-#979) where the actual question was always "is this slower than before", and answering it required manual before/after comparison.
Depends on #1028 — the
--report-jsonfile it adds is the baseline format.Proposal
baseline * (1 + tolerance/100).+12.4%,-3.1%) so improvements are visible too.Acceptance criteria
--baseline b.jsoncompares and prints a per-benchmark delta--baseline-updatewrites a file that a later--baselineaccepts, round-trip testedawkarithmetic, Bash 3.0 compatible, locale-independent (7 test failures on Bash 5.3 macOS (nix-shell) #912)docs/benchmarks.mdRepo checklist (agent)
printf -v, no+=append, nodeclare -A, no[[ ]], no${var,,}, no&>>, no${arr[-1]}. Expanding a possibly-empty array underset -uneeds${arr[@]+"${arr[@]}"}.src/main/test.sh(report-style flags needexport -n, seesrc/main/test.sh:188-196for why)bashunit::main::validate_config_or_exit(src/main/validate.sh:60) — unvalidated input used to run the wrong thing and exit 0 (Unknown options are silently ignored: a typo'd flag runs a different suite and exits 0 #871, --jobs with a non-integer value hangs on Bash 3.x and is silently ignored on Bash 4.3+ #873)src/config/env.shand a documented line in.env.example--helptext in the same block it belongs tocompletions/bashunit.bashandcompletions/_bashunit(anti-drift test feat(cli): bash and zsh completion scripts with an anti-drift test #778 fails otherwise)make sa,make lint,./bashunit tests/,./bashunit --parallel tests/. Never runshfmt -w.docs/command-line.md. Editingdocs/assertions.mdinvalidates thebashunit docacceptance snapshot — regenerate it.## Unreleased.tests/acceptance/fixtures/must not end in*test.sh.