🤔 Background
bashunit bench prints a file-level failure and then exits 0, so the error reaches a human reading the log and never reaches CI.
Same scenario, both subcommands — one good file plus one whose set_up_before_script fails:
$ bashunit bench benchmarks/✗ Error: Set up before scriptbench_works 1 1 6EXIT=0 <- bench
$ bashunit tests/EXIT=1 <- test
A benchmark file with a syntax error is worse, because nothing is reported as failed at all:
$ bashunit bench benchmarks/benchmarks/b_bench.sh: line 3: syntax error near unexpected token `{'Running benchmarks/b_bench.shBench ok [1/1] 6 msEXIT=0Every bench_ function after the syntax error is silently absent from the table, and the run is green. test exits 1 for the same file shape.
💡 Cause
Two gaps:
bashunit::runner::load_bench_files (src/runner/bench.sh) sources with no status check. The test loop in src/runner/discovery.sh checks the status and matches the captured stderr for syntax error / unexpected EOF, because bash reports a syntax error and carries on with a zero status.cmd_bench (src/main/run.sh) exits non-zero only for "No benchmarks found" and a baseline regression. It never consults the failure counter, so the existing set_up_before_script failure path — which already calls add_tests_failed — could not affect the verdict. It only looked correct when the failure left nothing to run, so "No benchmarks found" fired instead.
A benchmark function that merely returns non-zero should keep exiting 0: a benchmark measures time and has no assertion concept.
💡 Found by
Applying the pathological-fixture sweep from #1295/#1297/#1299/#1301 to the bench subcommand, which has its own runner and its own reports.
🤔 Background
bashunit benchprints a file-level failure and then exits 0, so the error reaches a human reading the log and never reaches CI.Same scenario, both subcommands — one good file plus one whose
set_up_before_scriptfails:A benchmark file with a syntax error is worse, because nothing is reported as failed at all:
Every
bench_function after the syntax error is silently absent from the table, and the run is green.testexits 1 for the same file shape.💡 Cause
Two gaps:
bashunit::runner::load_bench_files(src/runner/bench.sh) sources with no status check. The test loop insrc/runner/discovery.shchecks the status and matches the captured stderr forsyntax error/unexpected EOF, because bash reports a syntax error and carries on with a zero status.cmd_bench(src/main/run.sh) exits non-zero only for "No benchmarks found" and a baseline regression. It never consults the failure counter, so the existingset_up_before_scriptfailure path — which already callsadd_tests_failed— could not affect the verdict. It only looked correct when the failure left nothing to run, so "No benchmarks found" fired instead.A benchmark function that merely returns non-zero should keep exiting 0: a benchmark measures time and has no assertion concept.
💡 Found by
Applying the pathological-fixture sweep from #1295/#1297/#1299/#1301 to the
benchsubcommand, which has its own runner and its own reports.