You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
No reports of any kind. Results are printed by bashunit::benchmark::print_results (src/benchmark/results.sh:19) and then discarded, so a benchmark run in CI leaves nothing behind: no artifact to store, nothing to chart over time, nothing for a later run to compare against. Meanwhile bashunit test has JUnit, TAP, JSON, HTML and GHA reporters.
This blocks the obvious next step of failing a build on a performance regression, which is the whole point of running benchmarks in CI.
Proposal
--report-json <file> Machine-readable benchmark results
--report-junit <file> JUnit XML (so CI shows benchmarks alongside tests)
JSON shape, at minimum, per benchmark function: file, function, human-readable name, revs, its, per-iteration timings, average, min, max, median, and the @max_ms threshold with its pass/fail verdict. Plus a run-level block: total duration, timestamp, bashunit version, and enough host context (OS, bash version) to make two runs comparable.
Where to change
src/benchmark/results.sh:11add_result — the collection point.
src/benchmark/run.sh:5run_function, src/benchmark/annotations.sh for the @max_ms verdict.
bashunit bench --help and both completion scripts updated (parity test)
Tests in tests/unit/benchmark/ and tests/benchmark/
Repo checklist (agent)
TDD: RED → GREEN → REFACTOR. Write the failing test first.
Bash 3.0+ only: no printf -v, no += append, no declare -A, no [[ ]], no ${var,,}, no &>>, no ${arr[-1]}. Expanding a possibly-empty array under set -u needs ${arr[@]+"${arr[@]}"}.
A new CLI flag must be wired in all of these or a parity test fails:
parse in src/main/test.sh (report-style flags need export -n, see src/main/test.sh:188-196 for why)
Problem
bashunit benchis a second-class citizen. Its entire option set is:No reports of any kind. Results are printed by
bashunit::benchmark::print_results(src/benchmark/results.sh:19) and then discarded, so a benchmark run in CI leaves nothing behind: no artifact to store, nothing to chart over time, nothing for a later run to compare against. Meanwhilebashunit testhas JUnit, TAP, JSON, HTML and GHA reporters.This blocks the obvious next step of failing a build on a performance regression, which is the whole point of running benchmarks in CI.
Proposal
JSON shape, at minimum, per benchmark function: file, function, human-readable name,
revs,its, per-iteration timings, average, min, max, median, and the@max_msthreshold with its pass/fail verdict. Plus a run-level block: total duration, timestamp, bashunit version, and enough host context (OS, bash version) to make two runs comparable.Where to change
src/benchmark/results.sh:11add_result— the collection point.src/benchmark/run.sh:5run_function,src/benchmark/annotations.shfor the@max_msverdict.src/reports/writers where the shape allows; do not copy-paste a fourth parser (refactor(coverage,reports): share three copy-pasted parsers #971 consolidated three).src/main/bench.shfor the flags plus--help, and both completion scripts.Acceptance criteria
bashunit bench --report-json out.jsonwrites valid JSON (verify withjq)revs/its@max_msthreshold and verdict are present when the annotation is set--report-junitproduces XML that a CI test reporter acceptsenv LC_ALL=C awk, the 7 test failures on Bash 5.3 macOS (nix-shell) #912 workaround)bashunit bench --helpand both completion scripts updated (parity test)tests/unit/benchmark/andtests/benchmark/Repo checklist (agent)
printf -v, no+=append, nodeclare -A, no[[ ]], no${var,,}, no&>>, no${arr[-1]}. Expanding a possibly-empty array underset -uneeds${arr[@]+"${arr[@]}"}.src/main/test.sh(report-style flags needexport -n, seesrc/main/test.sh:188-196for why)bashunit::main::validate_config_or_exit(src/main/validate.sh:60) — unvalidated input used to run the wrong thing and exit 0 (Unknown options are silently ignored: a typo'd flag runs a different suite and exits 0 #871, --jobs with a non-integer value hangs on Bash 3.x and is silently ignored on Bash 4.3+ #873)src/config/env.shand a documented line in.env.example--helptext in the same block it belongs tocompletions/bashunit.bashandcompletions/_bashunit(anti-drift test feat(cli): bash and zsh completion scripts with an anti-drift test #778 fails otherwise)make sa,make lint,./bashunit tests/,./bashunit --parallel tests/. Never runshfmt -w.docs/command-line.md. Editingdocs/assertions.mdinvalidates thebashunit docacceptance snapshot — regenerate it.## Unreleased.tests/acceptance/fixtures/must not end in*test.sh.