From f7d9033090eb69d14447c30438e391a7e4951224 Mon Sep 17 00:00:00 2001 From: Chemaclass Date: Tue, 11 Aug 2026 14:27:07 +0200 Subject: [PATCH 01/10] docs(assertions): correct inverted examples, wrong summaries and undocumented semantics The assert_equals code-group was inverted: the documented test_success failed and test_failure passed. Verified by running both. Also: assert_true/assert_false summaries stated the opposite of the code, assert_files_not_equals reused the assert_files_equals sentence, the array assertions glob-match against the space-joined array rather than per element, assert_file_contains matches literally while assert_file_not_contains matches as a regex, JSON key assertions treat null/false as absent, the duration assertions error instead of skipping without awk, assert_between reports a usage error, assert_exec silently drops unrecognised flags, and assert_have_been_called_with eats a trailing numeric argument as nth. Directory examples used /home/user, which does not exist on macOS or CI; they now use bashunit::temp_dir. Four cross-references used an underscore anchor VitePress never generates. --- docs/assertions.md | 154 ++++++++++++++---- ...it_should_display_all_assert_docs.snapshot | 112 ++++++++++--- ...ould_display_filtered_assert_docs.snapshot | 6 +- 3 files changed, 215 insertions(+), 57 deletions(-) diff --git a/docs/assertions.md b/docs/assertions.md index 29ce8a79..bbc53c51 100644 --- a/docs/assertions.md +++ b/docs/assertions.md @@ -14,6 +14,18 @@ other helper does take the `bashunit::` prefix; see [Globals](/globals). Run `bashunit doc` to print this catalogue in your terminal, or `bashunit doc ` to narrow it (`bashunit doc json`). +Any assertion here also runs from the shell without a test file: +`bashunit assert contains "world" "hello world"`. See [Standalone](/standalone). + +`assert_same`, `assert_equals`, `assert_not_same`, `assert_not_equals` and the numeric +comparisons take an optional last argument used as the failure label: +`assert_same "1" "2" "my custom label"` reports `✗ Failed: my custom label`. + +The string assertions whose signature ends in `...` take a variadic value: every argument +after the first is joined with **newlines** and compared as one value. They accept no +trailing label override, so `assert_contains "zzz" "abc" "my label"` searches +`abc\nmy label` instead of relabelling the failure. + ## Quick reference | Group | Assertions | @@ -55,9 +67,10 @@ above. A purpose-built assertion is usually clearer still — `assert_directory_exists` rather than a hand-rolled `test -d`. -Reports an error if the argument result in a truthy value: `true` or `0`. +Reports an error **unless** the argument results in a truthy value: `true` or `0`, or a +command or function that exits `0`. -- [assert_false](#assert-false) is similar but different. +- [assert_false](#assert-false) is the inverse of this assertion and takes the same arguments. ::: code-group ```bash [Example] @@ -88,9 +101,10 @@ function mock_false() { ## assert_false > `assert_false bool|function|command [args...]` -Reports an error if the argument result in a falsy value: `false` or `1`. +Reports an error **unless** the argument results in a falsy value: `false` or `1`, or a +command or function that exits non-zero. -- [assert_true](#assert-true) is similar but different. +- [assert_true](#assert-true) is the inverse of this assertion and takes the same arguments. ::: code-group ```bash [Example] @@ -148,17 +162,17 @@ Reports an error if the two variables `expected` and `actual` are not equal igno ::: code-group ```bash [Example] function test_success() { - assert_equals "foo" "\e[31mfoo" + assert_equals "foo" $'\e[31mfoo' } function test_failure() { - assert_equals "\e[31mfoo" "\e[31mfoo" + assert_equals "foo" $'\e[31mbar' } ``` ::: ## assert_contains -> `assert_contains "needle" "haystack"` +> `assert_contains "needle" "haystack"...` Reports an error if `needle` is not a substring of `haystack`. @@ -214,10 +228,14 @@ function test_failure() { ::: ## assert_matches -> `assert_matches "pattern" "value"` +> `assert_matches "pattern" "value"...` Reports an error if `value` does not match the regular expression `pattern`. +`pattern` is an ERE evaluated by `grep -E`. If it does not match as written, the value is +retried with every newline replaced by a space, so a single-line pattern can match across +lines: `assert_matches 'one two'` matches the two-line value `one\ntwo`. + - [assert_not_matches](#assert-not-matches) is the inverse of this assertion and takes the same arguments. ::: code-group @@ -233,7 +251,7 @@ function test_failure() { ::: ## assert_string_starts_with -> `assert_string_starts_with "needle" "haystack"` +> `assert_string_starts_with "needle" "haystack"...` Reports an error if `haystack` does not starts with `needle`. @@ -252,7 +270,7 @@ function test_failure() { ::: ## assert_string_ends_with -> `assert_string_ends_with "needle" "haystack"` +> `assert_string_ends_with "needle" "haystack"...` Reports an error if `haystack` does not ends with `needle`. @@ -280,7 +298,7 @@ Reports an error if `value` does not match the `format` string. The format strin | `%d` | One or more digits | | `%i` | Signed integer (e.g. `+1`, `-42`) | | `%f` | Floating point number (e.g. `3.14`) | -| `%s` | One or more non-whitespace characters | +| `%s` | One or more characters other than a space (tabs and newlines match) | | `%x` | Hexadecimal (e.g. `ff00ab`) | | `%e` | Scientific notation (e.g. `1.5e10`) | | `%%` | Literal `%` character | @@ -301,10 +319,13 @@ function test_failure() { ::: ## assert_line_count -> `assert_line_count "count" "haystack"` +> `assert_line_count "count" "haystack"...` Reports an error if `haystack` does not contain `count` lines. +A literal `\n` (backslash followed by `n`) counts as a line break too, so +`assert_line_count 2 'one\ntwo'` passes even though the value holds no real newline. + ::: code-group ```bash [Example] function test_success() { @@ -345,7 +366,7 @@ function test_failure() { Reports an error if `actual` is not less than or equal to `expected`. -- [assert_greater_than](#assert-greater-or-equal-than) is the inverse of this assertion and takes the same arguments. +- [assert_greater_or_equal_than](#assert-greater-or-equal-than) is the counterpart of this assertion and takes the same arguments. ::: code-group ```bash [Example] @@ -411,6 +432,10 @@ function test_failure() { Reports an error if `actual` is outside the inclusive numeric range from `min` to `max`. Integers, decimals, and negative values are supported. `min` must not be greater than `max`. +A non-numeric argument, or `min` greater than `max`, is a **usage error**: the assertion +returns 2 and the test is reported as an Error rather than a failure, with a message such +as `assert_between expects min <= max, got '500' and '100'`. + - [assert_not_between](#assert-not-between) is the exact negation and takes the same arguments. ::: code-group @@ -432,6 +457,10 @@ function test_failure() { Reports an error if `actual` is inside the inclusive numeric range from `min` to `max`. Integers, decimals, and negative values are supported. `min` must not be greater than `max`. +A non-numeric argument, or `min` greater than `max`, is a **usage error**: the assertion +returns 2 and the test is reported as an Error rather than a failure, with a message such +as `assert_between expects min <= max, got '500' and '100'`. + - [assert_between](#assert-between) is the exact negation and takes the same arguments. ::: code-group @@ -453,6 +482,10 @@ Reports an error if `actual` is not within `delta` of `expected` (i.e. `|actual - expected| > delta`). Supports floating-point values. Useful for timing or measured values where exact equality is too strict. +The bound is inclusive, so `|actual - expected| == delta` passes. An operand that is not a +number, such as `1.2.3` or `5-3`, fails the assertion with `to all be numeric` instead of +being evaluated as an expression. A leading `+` is accepted. + ::: code-group ```bash [Example] function test_success() { @@ -480,6 +513,10 @@ Inputs are automatically converted to epoch seconds. Supported formats: You can mix formats in the same assertion (e.g., one epoch, one ISO). +Anything else, including an empty string, fails the assertion with +`Expected '' to be 'a valid date'` rather than being coerced to `0`. This applies to +all five date assertions. + ::: code-group ```bash [Example] function test_success() { @@ -500,7 +537,7 @@ function test_failure() { Reports an error if `actual` is not before `expected` (i.e. `actual` must be less than `expected`). -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -519,7 +556,7 @@ function test_failure() { Reports an error if `actual` is not after `expected` (i.e. `actual` must be greater than `expected`). -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -538,7 +575,7 @@ function test_failure() { Reports an error if `actual` does not fall between `from` and `to` (inclusive). -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -557,7 +594,10 @@ function test_failure() { Reports an error if `actual` is not within `delta` seconds of `expected`. -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +`delta` is required: omitting it produces a shell error reported as an Error, not an +assertion failure. + +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -634,6 +674,11 @@ Use `--stdin` to feed input into interactive commands (e.g. commands using Use `--stdout-contains` / `--stdout-not-contains` (and the `stderr-*` variants) for substring matching when you don't want to assert against the full output. +Unrecognised arguments are silently dropped, so a mistyped flag such as +`--stdout-contain` checks nothing and the assertion passes on exit status alone. Extra +words after the command are dropped too: put arguments inside the command string, +`assert_exec "echo hello" --stdout "hello"`. + ::: code-group ```bash [Example] function sample() { @@ -697,7 +742,12 @@ function test_failure() { ## assert_array_contains > `assert_array_contains "needle" "haystack"` -Reports an error if `needle` is not an element of `haystack`. +Reports an error if `needle` is not found in `haystack`. + +`needle` is matched as a **substring of the array joined with spaces**, not element by +element, so `assert_array_contains "oob" foobar baz` and +`assert_array_contains "foo bar" foo bar baz` both pass. Use +[assert_arrays_equal](#assert-arrays-equal) when you need exact element comparison. - [assert_array_not_contains](#assert-array-not-contains) is the inverse of this assertion and takes the same arguments. @@ -960,7 +1010,9 @@ function test_failure() { ## assert_file_contains > `assert_file_contains "file" "search"` -Reports an error if `file` does not contains the search string. +Reports an error if `file` does not contain the search string. + +`search` is matched **literally** (`grep -F`); regex metacharacters have no special meaning. - [assert_file_not_contains](#assert-file-not-contains) is the inverse of this assertion and takes the same arguments. @@ -1158,8 +1210,8 @@ Reports an error if `directory` is not an empty directory. ::: code-group ```bash [Example] function test_success() { - local directory="/home/user/empty_directory" - mkdir "$directory" + local directory + directory="$(bashunit::temp_dir)" assert_is_directory_empty "$directory" } @@ -1188,7 +1240,8 @@ function test_success() { } function test_failure() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -r "$directory" assert_is_directory_readable "$directory" @@ -1212,7 +1265,8 @@ function test_success() { } function test_failure() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -w "$directory" assert_is_directory_writable "$directory" @@ -1299,7 +1353,7 @@ function test_failure() { ::: ## assert_not_contains -> `assert_not_contains "needle" "haystack"` +> `assert_not_contains "needle" "haystack"...` Reports an error if `needle` is a substring of `haystack`. @@ -1318,7 +1372,7 @@ function test_failure() { ::: ## assert_string_not_starts_with -> `assert_string_not_starts_with "needle" "haystack"` +> `assert_string_not_starts_with "needle" "haystack"...` Reports an error if `haystack` does starts with `needle`. @@ -1337,7 +1391,7 @@ function test_failure() { ::: ## assert_string_not_ends_with -> `assert_string_not_ends_with "needle" "haystack"` +> `assert_string_not_ends_with "needle" "haystack"...` Reports an error if `haystack` does ends with `needle`. @@ -1375,7 +1429,7 @@ function test_failure() { ::: ## assert_not_matches -> `assert_not_matches "pattern" "value"` +> `assert_not_matches "pattern" "value"...` Reports an error if `value` matches the regular expression `pattern`. @@ -1415,7 +1469,11 @@ function test_failure() { ## assert_array_not_contains > `assert_array_not_contains "needle" "haystack"` -Reports an error if `needle` is an element of `haystack`. +Reports an error if `needle` is found in `haystack`. + +`needle` is matched as a **substring of the array joined with spaces**, not element by +element, so `assert_array_not_contains "foo bar" foo bar` fails even though no single +element is `foo bar`. - [assert_array_contains](#assert-array-contains) is the inverse of this assertion and takes the same arguments. @@ -1467,6 +1525,10 @@ function test_failed() { Reports an error if `file` contains the search string. +`search` is matched as a **basic regular expression** (`grep`), unlike +[assert_file_contains](#assert-file-contains) which matches literally, so +`assert_file_not_contains file 'a.c'` fails on a file containing `abc`. + - [assert_file_contains](#assert-file-contains) is the inverse of this assertion and takes the same arguments. ::: code-group @@ -1526,8 +1588,8 @@ function test_success() { } function test_failure() { - local directory="/home/user/empty_directory" - mkdir "$directory" + local directory + directory="$(bashunit::temp_dir)" assert_is_directory_not_empty "$directory" } @@ -1544,7 +1606,8 @@ Reports an error if `directory` is readable. ::: code-group ```bash [Example] function test_success() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -r "$directory" assert_is_directory_not_readable "$directory" @@ -1568,7 +1631,8 @@ Reports an error if `directory` is writable. ::: code-group ```bash [Example] function test_success() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -w "$directory" assert_is_directory_not_writable "$directory" @@ -1586,7 +1650,7 @@ function test_failure() { ## assert_files_not_equals > `assert_files_not_equals "expected" "actual"` -Reports an error if `expected` and `actual` are not equals. +Reports an error if `expected` and `actual` have the same contents. - [assert_files_equals](#assert-files-equals) is the inverse of this assertion and takes the same arguments. @@ -1627,6 +1691,10 @@ function test_failure() { Reports an error if `key` does not exist in the JSON string. Uses [jq](https://jqlang.github.io/jq/) syntax for key paths. Requires `jq` to be installed; if missing the test is skipped. +A key whose value is `null` or `false` is reported as missing, because `jq -e` treats both +as absent. `0` and `""` are fine. To assert a false or null value, use +`assert_json_contains ".flag" "false" "$json"`. + ::: code-group ```bash [Example] function test_success() { @@ -1645,6 +1713,9 @@ function test_failure() { Reports an error if `key` does not exist in the JSON string or its value does not equal `expected`. Uses [jq](https://jqlang.github.io/jq/) syntax for key paths. Requires `jq` to be installed; if missing the test is skipped. +A key whose value is `null` or `false` is reported as missing, the same guard +[assert_json_key_exists](#assert-json-key-exists) uses. + ::: code-group ```bash [Example] function test_success() { @@ -1681,6 +1752,10 @@ function test_failure() { Reports an error if `command` takes longer than `threshold_ms` milliseconds to execute. Uses the framework's portable clock internally. +Requires `awk`, plus `bc` or `awk` for the arithmetic. Unlike the JSON assertions, the +duration assertions do **not** skip when the tool is missing: the test is reported as an +Error. + ::: code-group ```bash [Example] function test_success() { @@ -1798,6 +1873,11 @@ function test_colored_render_modes() { Reports an error if the spied `command` was never called. Requires `bashunit::spy command` first — see [Test doubles](/test-doubles). +Every `assert_have_been_called*` and `assert_not_called` fails with +`was never registered as a spy` when the name was never passed to `bashunit::spy`, +including `assert_not_called`, which does **not** pass for an unspied name. Spies are +cleared between tests, so spy inside the test that asserts on it. + ::: code-group ```bash [Example] function test_success() { @@ -1818,7 +1898,12 @@ function test_failure() { ## assert_have_been_called_with > `assert_have_been_called_with "command" "expected_args" [nth]` -Reports an error if the spied `command` was not called with `expected_args`. Checks the **last** call unless a trailing all-digits `nth` selects a specific one; the failure names the call it compared. To match any call, use [assert_have_been_called_with_any](#assert-have-been-called-with-any). +Reports an error if the spied `command` was not called with `expected_args`. Checks the **last** call unless a trailing all-digits `nth` selects a specific one; the failure names the call it compared. +Because `nth` is detected as a trailing all-digits argument, an expectation whose last +argument is a number is read as the selector: `assert_have_been_called_with git commit -m 5` +compares `commit -m` against call 5. Pass the expectation as one quoted string, +`assert_have_been_called_with git "commit -m 5"`, or use +[assert_have_been_called_with_args](#assert-have-been-called-with-args), which has no `nth`. To match any call, use [assert_have_been_called_with_any](#assert-have-been-called-with-any). Note the argument order: the spy comes first here, but *second* in [assert_have_been_called_times](#assert-have-been-called-times). @@ -2045,3 +2130,4 @@ function test_failure() { - [Test doubles](/test-doubles) — mocks and spies for isolated tests - [Data providers](/data-providers) — run the same assertions over many inputs - [Globals](/globals) — `bashunit::` helper functions +- [Standalone](/standalone) — run these assertions straight from the command line diff --git a/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_all_assert_docs.snapshot b/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_all_assert_docs.snapshot index a4274136..0b80acbb 100644 --- a/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_all_assert_docs.snapshot +++ b/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_all_assert_docs.snapshot @@ -9,9 +9,10 @@ Pass a command with its arguments as separate arguments: -------------- > `assert_false bool|function|command args...` -Reports an error if the argument result in a falsy value: `false` or `1`. +Reports an error **unless** the argument results in a falsy value: `false` or `1`, or a +command or function that exits non-zero. -- assert_true is similar but different. +- assert_true is the inverse of this assertion and takes the same arguments. ## assert_same @@ -35,7 +36,7 @@ Reports an error if the two variables `expected` and `actual` are not equal igno ## assert_contains -------------- -> `assert_contains "needle" "haystack"` +> `assert_contains "needle" "haystack"...` Reports an error if `needle` is not a substring of `haystack`. @@ -61,16 +62,20 @@ Reports an error if `actual` is not empty. ## assert_matches -------------- -> `assert_matches "pattern" "value"` +> `assert_matches "pattern" "value"...` Reports an error if `value` does not match the regular expression `pattern`. +`pattern` is an ERE evaluated by `grep -E`. If it does not match as written, the value is +retried with every newline replaced by a space, so a single-line pattern can match across +lines: `assert_matches 'one two'` matches the two-line value `one\ntwo`. + - assert_not_matches is the inverse of this assertion and takes the same arguments. ## assert_string_starts_with -------------- -> `assert_string_starts_with "needle" "haystack"` +> `assert_string_starts_with "needle" "haystack"...` Reports an error if `haystack` does not starts with `needle`. @@ -79,7 +84,7 @@ Reports an error if `haystack` does not starts with `needle`. ## assert_string_ends_with -------------- -> `assert_string_ends_with "needle" "haystack"` +> `assert_string_ends_with "needle" "haystack"...` Reports an error if `haystack` does not ends with `needle`. @@ -97,7 +102,7 @@ Reports an error if `value` does not match the `format` string. The format strin | `%d` | One or more digits | | `%i` | Signed integer (e.g. `+1`, `-42`) | | `%f` | Floating point number (e.g. `3.14`) | -| `%s` | One or more non-whitespace characters | +| `%s` | One or more characters other than a space (tabs and newlines match) | | `%x` | Hexadecimal (e.g. `ff00ab`) | | `%e` | Scientific notation (e.g. `1.5e10`) | | `%%` | Literal `%` character | @@ -107,10 +112,13 @@ Reports an error if `value` does not match the `format` string. The format strin ## assert_line_count -------------- -> `assert_line_count "count" "haystack"` +> `assert_line_count "count" "haystack"...` Reports an error if `haystack` does not contain `count` lines. +A literal `\n` (backslash followed by `n`) counts as a line break too, so +`assert_line_count 2 'one\ntwo'` passes even though the value holds no real newline. + ## assert_less_than -------------- @@ -127,7 +135,7 @@ Reports an error if `actual` is not less than `expected`. Reports an error if `actual` is not less than or equal to `expected`. -- assert_greater_than is the inverse of this assertion and takes the same arguments. +- assert_greater_or_equal_than is the counterpart of this assertion and takes the same arguments. ## assert_greater_than @@ -155,6 +163,10 @@ Reports an error if `actual` is not greater than or equal to `expected`. Reports an error if `actual` is outside the inclusive numeric range from `min` to `max`. Integers, decimals, and negative values are supported. `min` must not be greater than `max`. +A non-numeric argument, or `min` greater than `max`, is a **usage error**: the assertion +returns 2 and the test is reported as an Error rather than a failure, with a message such +as `assert_between expects min <= max, got '500' and '100'`. + - assert_not_between is the exact negation and takes the same arguments. @@ -165,6 +177,10 @@ Integers, decimals, and negative values are supported. `min` must not be greater Reports an error if `actual` is inside the inclusive numeric range from `min` to `max`. Integers, decimals, and negative values are supported. `min` must not be greater than `max`. +A non-numeric argument, or `min` greater than `max`, is a **usage error**: the assertion +returns 2 and the test is reported as an Error rather than a failure, with a message such +as `assert_between expects min <= max, got '500' and '100'`. + - assert_between is the exact negation and takes the same arguments. @@ -176,6 +192,10 @@ Reports an error if `actual` is not within `delta` of `expected` (i.e. `|actual - expected| > delta`). Supports floating-point values. Useful for timing or measured values where exact equality is too strict. +The bound is inclusive, so `|actual - expected| == delta` passes. An operand that is not a +number, such as `1.2.3` or `5-3`, fails the assertion with `to all be numeric` instead of +being evaluated as an expression. A leading `+` is accepted. + ## assert_date_equals -------------- @@ -193,6 +213,10 @@ Inputs are automatically converted to epoch seconds. Supported formats: You can mix formats in the same assertion (e.g., one epoch, one ISO). +Anything else, including an empty string, fails the assertion with +`Expected '' to be 'a valid date'` rather than being coerced to `0`. This applies to +all five date assertions. + ## assert_date_before -------------- @@ -200,7 +224,7 @@ You can mix formats in the same assertion (e.g., one epoch, one ISO). Reports an error if `actual` is not before `expected` (i.e. `actual` must be less than `expected`). -Inputs are automatically converted to epoch seconds. See assert_date_equals(#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See assert_date_equals for supported formats. ## assert_date_after @@ -209,7 +233,7 @@ Inputs are automatically converted to epoch seconds. See assert_date_equals(#ass Reports an error if `actual` is not after `expected` (i.e. `actual` must be greater than `expected`). -Inputs are automatically converted to epoch seconds. See assert_date_equals(#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See assert_date_equals for supported formats. ## assert_date_within_range @@ -218,7 +242,7 @@ Inputs are automatically converted to epoch seconds. See assert_date_equals(#ass Reports an error if `actual` does not fall between `from` and `to` (inclusive). -Inputs are automatically converted to epoch seconds. See assert_date_equals(#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See assert_date_equals for supported formats. ## assert_date_within_delta @@ -227,7 +251,10 @@ Inputs are automatically converted to epoch seconds. See assert_date_equals(#ass Reports an error if `actual` is not within `delta` seconds of `expected`. -Inputs are automatically converted to epoch seconds. See assert_date_equals(#assert_date_equals) for supported formats. +`delta` is required: omitting it produces a shell error reported as an Error, not an +assertion failure. + +Inputs are automatically converted to epoch seconds. See assert_date_equals for supported formats. ## assert_exit_code @@ -262,6 +289,11 @@ Use `--stdin` to feed input into interactive commands (e.g. commands using Use `--stdout-contains` / `--stdout-not-contains` (and the `stderr-*` variants) for substring matching when you don't want to assert against the full output. +Unrecognised arguments are silently dropped, so a mistyped flag such as +`--stdout-contain` checks nothing and the assertion passes on exit status alone. Extra +words after the command are dropped too: put arguments inside the command string, +`assert_exec "echo hello" --stdout "hello"`. + ## assert_arrays_equal -------------- @@ -276,7 +308,12 @@ Use `--` to separate the expected array from the actual array. -------------- > `assert_array_contains "needle" "haystack"` -Reports an error if `needle` is not an element of `haystack`. +Reports an error if `needle` is not found in `haystack`. + +`needle` is matched as a **substring of the array joined with spaces**, not element by +element, so `assert_array_contains "oob" foobar baz` and +`assert_array_contains "foo bar" foo bar baz` both pass. Use +assert_arrays_equal when you need exact element comparison. - assert_array_not_contains is the inverse of this assertion and takes the same arguments. @@ -381,7 +418,9 @@ Reports an error if `file` does not exists, or it is a directory. -------------- > `assert_file_contains "file" "search"` -Reports an error if `file` does not contains the search string. +Reports an error if `file` does not contain the search string. + +`search` is matched **literally** (`grep -F`); regex metacharacters have no special meaning. - assert_file_not_contains is the inverse of this assertion and takes the same arguments. @@ -506,7 +545,7 @@ Reports an error if the two variables `expected` and `actual` are the same value ## assert_not_contains -------------- -> `assert_not_contains "needle" "haystack"` +> `assert_not_contains "needle" "haystack"...` Reports an error if `needle` is a substring of `haystack`. @@ -515,7 +554,7 @@ Reports an error if `needle` is a substring of `haystack`. ## assert_string_not_starts_with -------------- -> `assert_string_not_starts_with "needle" "haystack"` +> `assert_string_not_starts_with "needle" "haystack"...` Reports an error if `haystack` does starts with `needle`. @@ -524,7 +563,7 @@ Reports an error if `haystack` does starts with `needle`. ## assert_string_not_ends_with -------------- -> `assert_string_not_ends_with "needle" "haystack"` +> `assert_string_not_ends_with "needle" "haystack"...` Reports an error if `haystack` does ends with `needle`. @@ -542,7 +581,7 @@ Reports an error if `actual` is empty. ## assert_not_matches -------------- -> `assert_not_matches "pattern" "value"` +> `assert_not_matches "pattern" "value"...` Reports an error if `value` matches the regular expression `pattern`. @@ -562,7 +601,11 @@ Reports an error if `value` matches the `format` string. See assert_string_match -------------- > `assert_array_not_contains "needle" "haystack"` -Reports an error if `needle` is an element of `haystack`. +Reports an error if `needle` is found in `haystack`. + +`needle` is matched as a **substring of the array joined with spaces**, not element by +element, so `assert_array_not_contains "foo bar" foo bar` fails even though no single +element is `foo bar`. - assert_array_contains is the inverse of this assertion and takes the same arguments. @@ -582,6 +625,10 @@ Reports an error if `file` does exists. Reports an error if `file` contains the search string. +`search` is matched as a **basic regular expression** (`grep`), unlike +assert_file_contains which matches literally, so +`assert_file_not_contains file 'a.c'` fails on a file containing `abc`. + - assert_file_contains is the inverse of this assertion and takes the same arguments. @@ -625,7 +672,7 @@ Reports an error if `directory` is writable. -------------- > `assert_files_not_equals "expected" "actual"` -Reports an error if `expected` and `actual` are not equals. +Reports an error if `expected` and `actual` have the same contents. - assert_files_equals is the inverse of this assertion and takes the same arguments. @@ -636,6 +683,10 @@ Reports an error if `expected` and `actual` are not equals. Reports an error if `key` does not exist in the JSON string. Uses jq(https://jqlang.github.io/jq/) syntax for key paths. Requires `jq` to be installed; if missing the test is skipped. +A key whose value is `null` or `false` is reported as missing, because `jq -e` treats both +as absent. `0` and `""` are fine. To assert a false or null value, use +`assert_json_contains ".flag" "false" "$json"`. + ## assert_json_contains -------------- @@ -643,6 +694,9 @@ Reports an error if `key` does not exist in the JSON string. Uses jq(https://jql Reports an error if `key` does not exist in the JSON string or its value does not equal `expected`. Uses jq(https://jqlang.github.io/jq/) syntax for key paths. Requires `jq` to be installed; if missing the test is skipped. +A key whose value is `null` or `false` is reported as missing, the same guard +assert_json_key_exists uses. + ## assert_json_equals -------------- @@ -657,6 +711,10 @@ Reports an error if the two JSON strings are not structurally equal. Key order i Reports an error if `command` takes longer than `threshold_ms` milliseconds to execute. Uses the framework's portable clock internally. +Requires `awk`, plus `bc` or `awk` for the arithmetic. Unlike the JSON assertions, the +duration assertions do **not** skip when the tool is missing: the test is reported as an +Error. + ## assert_duration_less_than -------------- @@ -710,12 +768,22 @@ Named version of assert_match_snapshot_ignore_colors. ANSI escape sequences are Reports an error if the spied `command` was never called. Requires `bashunit::spy command` first — see Test doubles(/test-doubles). +Every `assert_have_been_called*` and `assert_not_called` fails with +`was never registered as a spy` when the name was never passed to `bashunit::spy`, +including `assert_not_called`, which does **not** pass for an unspied name. Spies are +cleared between tests, so spy inside the test that asserts on it. + ## assert_have_been_called_with -------------- > `assert_have_been_called_with "command" "expected_args" nth` -Reports an error if the spied `command` was not called with `expected_args`. Checks the **last** call unless a trailing all-digits `nth` selects a specific one; the failure names the call it compared. To match any call, use assert_have_been_called_with_any. +Reports an error if the spied `command` was not called with `expected_args`. Checks the **last** call unless a trailing all-digits `nth` selects a specific one; the failure names the call it compared. +Because `nth` is detected as a trailing all-digits argument, an expectation whose last +argument is a number is read as the selector: `assert_have_been_called_with git commit -m 5` +compares `commit -m` against call 5. Pass the expectation as one quoted string, +`assert_have_been_called_with git "commit -m 5"`, or use +assert_have_been_called_with_args, which has no `nth`. To match any call, use assert_have_been_called_with_any. Note the argument order: the spy comes first here, but *second* in assert_have_been_called_times. diff --git a/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_filtered_assert_docs.snapshot b/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_filtered_assert_docs.snapshot index 957e0848..dd45cb34 100644 --- a/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_filtered_assert_docs.snapshot +++ b/tests/acceptance/snapshots/bashunit_test_sh.test_bashunit_should_display_filtered_assert_docs.snapshot @@ -23,6 +23,10 @@ Inputs are automatically converted to epoch seconds. Supported formats: You can mix formats in the same assertion (e.g., one epoch, one ISO). +Anything else, including an empty string, fails the assertion with +`Expected '' to be 'a valid date'` rather than being coerced to `0`. This applies to +all five date assertions. + ## assert_files_equals -------------- @@ -46,7 +50,7 @@ Reports an error if the two variables `expected` and `actual` are equal ignoring -------------- > `assert_files_not_equals "expected" "actual"` -Reports an error if `expected` and `actual` are not equals. +Reports an error if `expected` and `actual` have the same contents. - assert_files_equals is the inverse of this assertion and takes the same arguments. From 3ee744bc90f205e978044a89e0f1ca13ad26e0cd Mon Sep 17 00:00:00 2001 From: Chemaclass Date: Tue, 11 Aug 2026 14:29:23 +0200 Subject: [PATCH 02/10] docs(config): bring the settings reference back to parity with env.sh Adds the 17 registered settings the page never documented: ORDER_BY, REPEAT, FAIL_ON_FLAKY, EXCLUDE_FILTER, CHANGED, CHANGED_REF, SHARD_INDEX/TOTAL, LIST_TESTS, LIST_FORMAT, REPORT_MD, GHA_ANNOTATIONS, SNAPSHOT_REPORT_UNUSED, COVERAGE_REPORT_HTML, COVERAGE_DIFF, the three coverage detail blocks, and BENCH_MODE. Corrections: DEFAULT_PATH defaults to tests and not empty; an empty BASHUNIT_COVERAGE_REPORT does not disable the file, --no-coverage-report does; parallel does not randomize; -vvv is not the counterpart of --simple; NO_COLOR is the external standard while BASHUNIT_NO_COLOR is the setting; coverage colour thresholds are inclusive; only 43 of 66 settings have an unprefixed alias, the rest are prefix-only by design. The precedence list gains the --env/--boot file, which overrides .env and the ambient environment, and the section now says .env is sourced as shell while .bashunitrc is parsed as KEY=value. The coverage env-var list existed twice and had drifted; configuration.md is now the single owner and docs/coverage.md links to it. coverage.md also stated that path matching decides what is tracked, when only executed files are ever reported (#1053). --- docs/configuration.md | 212 +++++++++++++++++++++++++++++++++++++----- docs/coverage.md | 48 ++++------ 2 files changed, 206 insertions(+), 54 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 38fbcca1..4012a886 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -7,9 +7,12 @@ description: "Configure bashunit with environment variables and config files to Environment variables and config files control **bashunit** behavior across your project. It serves to configure the behavior of bashunit in your project. -You need to create a `.env` file in the root directory, -but you can give it another name if you pass it as an argument to the command with -`--env` [option](/command-line#environment-bootstrap). +`.env` and `.bashunitrc` are project files, always read from the working directory. + +`-e, --env` (alias `--boot`) loads an **additional** bootstrap file whose assignments +override them; it does not rename `.env`. See +[the option](/command-line#environment-bootstrap). `--skip-env-file` is the only way to +stop `.env` and `.bashunitrc` from being read at all. ## Config file (.bashunitrc) @@ -27,10 +30,16 @@ It is meant for committing sensible project defaults. Precedence, from highest to lowest: 1. CLI flags (e.g. `--simple`) -2. `.env` entries that have a value -3. Environment variables -4. `.bashunitrc` -5. Built-in defaults +2. The file given to `-e, --env, --boot` +3. `.env` entries that have a value +4. Environment variables +5. `.bashunitrc` +6. Built-in defaults + +The `--env` file is sourced during flag parsing, so it overrides `.env`, `.bashunitrc` and +the ambient environment, and it loses only to flags written **after** it on the command +line: `bashunit --simple --env custom.env` uses the file's value, `bashunit --env custom.env --simple` +uses the flag. Unlike `.env`, an empty entry in that file does wipe a value. An entry left **empty** in `.env` means "not configured here" and does not override the environment, so `BASHUNIT_OUTPUT_FORMAT=tap ./bashunit` keeps @@ -41,11 +50,17 @@ committed project config is for. `.bashunitrc` only fills values that are not already set, so anything above it always wins. `--skip-env-file` skips both `.env` and `.bashunitrc`. +The two files are read differently, which is why they behave differently: `.env` is +**sourced** as a shell script under `allexport`, so it can hold arbitrary shell and an +empty entry is unconditional (hence the preservation rule above), while `.bashunitrc` is +parsed as literal `KEY=value` lines and only fills names that are not already set. + ## Default path > `BASHUNIT_DEFAULT_PATH=directory|file` -Specifies the `directory` or `file` containing the tests to be run. `empty` by default. +Specifies the `directory` or `file` containing the tests to be run. `tests` by default, +so a run with no path argument searches `tests/` for files ending in `test.sh`. If a directory is specified, it will execute tests within files ending in `test.sh`. When running benchmarks (`bashunit bench`), the same path is used to search for files ending in `bench.sh`. @@ -71,9 +86,12 @@ BASHUNIT_DEFAULT_PATH=tests/**/*_test.sh Enables simplified output to the console. `false` by default. -Verbose is the default output, but it can be overridden by the environment configuration. +Detailed output is the default, but it can be overridden by the environment configuration. -Similar as using `-s|--simple | -vvv|--detailed` option on the [command line](/command-line#output-style). +Similar as using `-s|--simple` / `--detailed` on the [command line](/command-line#output-style). + +This is a different setting from [`BASHUNIT_VERBOSE`](#verbose): `-vvv|--verbose` adds the +execution-details block and does not change the result style. ::: code-group ```bash [Simple output] @@ -85,7 +103,7 @@ BASHUNIT_SIMPLE_OUTPUT=true ::: ::: code-group -```[Verbose output] +```[Detailed output] Running tests/functional/logic_test.sh ✓ Passed: Other way of using the exit code ✓ Passed: Should validate a non ok exit code @@ -101,7 +119,14 @@ BASHUNIT_SIMPLE_OUTPUT=false > `BASHUNIT_PARALLEL_RUN=true|false` -Runs the tests in child processes with randomized execution, which may improve overall testing speed, especially for larger test suites. +Runs the tests in child processes, one worker per test, which may improve overall testing +speed, especially for larger test suites. `false` by default. + +Dispatch keeps definition order, but **completion** order is nondeterministic, so tests +must not depend on each other. Parallel never shuffles: shuffling is opt-in through +[`BASHUNIT_RANDOM_ORDER`](#random-order) or `BASHUNIT_ORDER_BY=random`. Cap the concurrency +with [`BASHUNIT_PARALLEL_JOBS`](#parallel-jobs), and use `--no-parallel` to opt out of a +configured parallel run. ::: warning Parallel execution is supported on **macOS**, **Ubuntu**, **Alpine**, and @@ -179,6 +204,25 @@ BASHUNIT_RETRY=0 ``` ::: +## Repeat + +> `BASHUNIT_REPEAT=` + +Run each selected test `n` times so flakiness surfaces before CI hits it. `1` by default. +The test is reported once with the aggregate outcome, and a failure names the iteration it +happened on. Repeat wraps [`BASHUNIT_RETRY`](#retry), not the other way round. + +Similar as using `--repeat` option on the [command line](/command-line#repeat). + +## Fail on flaky + +> `BASHUNIT_FAIL_ON_FLAKY=true|false` + +Turn a run red when a test passed only after a retry. `false` by default, so a flaky test +stays inside the pass total and the exit code is unchanged. + +Similar as using `--fail-on-flaky` option on the [command line](/command-line#fail-on-flaky). + ## Random order > `BASHUNIT_RANDOM_ORDER=true|false` and `BASHUNIT_SEED=` @@ -201,6 +245,51 @@ BASHUNIT_RANDOM_ORDER=false ``` ::: +## Execution order + +> `BASHUNIT_ORDER_BY=defined|defects|random` + +Execution order. `defined` by default (definition order). `defects` runs the last run's +failures first and still runs the whole suite. `random` is the same mode as +[`BASHUNIT_RANDOM_ORDER=true`](#random-order). + +Similar as using `--order-by` option on the [command line](/command-line#order-by). + +## Exclude filter + +> `BASHUNIT_EXCLUDE_FILTER=name` + +Skip tests whose name matches. Empty by default. It wins over a `--filter` match. + +Similar as using `--exclude-filter` option on the [command line](/command-line#exclude-filter). + +## Changed files only + +> `BASHUNIT_CHANGED=true|false` and `BASHUNIT_CHANGED_REF=` + +Run only the test files git reports as changed. `false` by default. +`BASHUNIT_CHANGED_REF` is empty by default, which means `origin/HEAD`, then `HEAD`. + +Similar as using `--changed` option on the [command line](/command-line#changed). + +## Shard + +> `BASHUNIT_SHARD_INDEX=` and `BASHUNIT_SHARD_TOTAL=` + +Run shard `i` of `n` to split the suite across runners. Both are empty (disabled) by +default, and sharding needs both. + +Similar as using `--shard` option on the [command line](/command-line#shard). + +## List tests + +> `BASHUNIT_LIST_TESTS=true|false` and `BASHUNIT_LIST_FORMAT=text|json` + +Print the tests a run would execute and exit without running them. `false` and `text` by +default. + +Similar as using `--list` / `--list-format` options on the [command line](/command-line#list). + ## Snapshot update > `BASHUNIT_SNAPSHOT_UPDATE=true|false` @@ -243,6 +332,15 @@ BASHUNIT_SNAPSHOT_CREATE=true ``` ::: +## Snapshot report unused + +> `BASHUNIT_SNAPSHOT_REPORT_UNUSED=true|false` + +List the snapshot files no test resolved. `false` by default. Full runs only, and it deletes +nothing. + +Similar as using `--snapshot-report-unused` option on the [command line](/command-line#snapshot-report-unused). + ## Rerun failed > `BASHUNIT_RERUN_FAILED=true|false` @@ -507,6 +605,25 @@ The report destination is checked before the suite runs — an unwritable path f immediately instead of after a passing run. See [Invalid input](/command-line#invalid-input). ::: +## Report Markdown + +> `BASHUNIT_REPORT_MD=file` + +Write a Markdown run summary: verdict, counts table, failures with their message, plus +coverage and slowest tests when those ran. Empty by default. Inside GitHub Actions it is +also appended to `$GITHUB_STEP_SUMMARY`, for the outermost run only. + +Similar as using `--report-md` option on the [command line](/command-line#report-md). + +## GitHub Actions annotations + +> `BASHUNIT_GHA_ANNOTATIONS=auto|always|never` + +Controls GitHub Actions annotations on stdout. `auto` by default: on inside GitHub Actions, +quiet everywhere else. Never emitted under `--output tap`. + +Similar as using `--gha-annotations` option on the [command line](/command-line#gha-annotations). + ## Bootstrap > `BASHUNIT_BOOTSTRAP=file` @@ -577,7 +694,7 @@ bashunit::log "I am tracing something..." bashunit::log "error" "an" "error" "message" bashunit::log "warning" "different log level messages!" ``` -```bash [Output: out.log] +```[Output: dev.log] 2024-10-03 21:27:23 [INFO]: I am tracing something... #tests/sample.sh:11 2024-10-03 21:27:23 [ERROR]: an error message #tests/sample.sh:27 2024-10-03 21:27:24 [WARNING]: different log level messages! #tests/sample.sh:21 @@ -589,6 +706,13 @@ quickly `tail -f` it while the tests run. > All internal messages emitted by bashunit are prefixed with `[INTERNAL]`. > You can toggle internal messages with `BASHUNIT_INTERNAL_LOG=true|false`. +## Bench mode + +> `BASHUNIT_BENCH_MODE=true|false` + +Set by the `bashunit bench` subcommand and not meant to be configured by hand. `false` by +default. + ## Verbose > `BASHUNIT_VERBOSE=bool` @@ -738,9 +862,12 @@ BASHUNIT_NO_DIFF=true ## Color output -> `NO_COLOR=1` +> `BASHUNIT_NO_COLOR=true|false` + +Disables ANSI color codes in output. `false` by default. -Disables ANSI color codes in output. Follows the [no-color.org](https://no-color.org) standard. +`NO_COLOR` is the honored external standard: a non-empty `NO_COLOR` forces +`BASHUNIT_NO_COLOR=true`. Follows [no-color.org](https://no-color.org). When set to any value, bashunit will output plain text without color formatting. @@ -872,18 +999,49 @@ BASHUNIT_COVERAGE_EXCLUDE=tests/*,vendor/*,*_test.sh,*_mock.sh Path for the LCOV format coverage report. `coverage/lcov.info` by default. -Set to empty string to disable file generation (console report only). +An empty entry means "not configured", so the default path still applies. To skip the file +and keep the console report only, use the `--no-coverage-report` flag. ::: code-group ```bash [.env] # Custom path BASHUNIT_COVERAGE_REPORT=reports/coverage.lcov - -# Disable file output -BASHUNIT_COVERAGE_REPORT= +``` +```bash [Console only] +bashunit tests/ --coverage --no-coverage-report ``` ::: +### Coverage report HTML + +> `BASHUNIT_COVERAGE_REPORT_HTML=dir` + +Directory for the browsable HTML coverage report. Empty by default (not generated). + +Similar as using `--coverage-report-html` option on the [command line](/command-line#coverage). + +### Coverage diff + +> `BASHUNIT_COVERAGE_DIFF=` + +Restrict the console coverage report to the lines changed since ``. Empty by default. +`BASHUNIT_COVERAGE_MIN` then gates on that diff percentage instead of the whole-file one. +LCOV and HTML stay whole-file. + +Similar as using `--coverage-diff` option on the [command line](/command-line#coverage). + +### Coverage detail blocks + +> `BASHUNIT_COVERAGE_SHOW_FUNCTIONS=true|false` +> +> `BASHUNIT_COVERAGE_SHOW_UNCOVERED=true|false` +> +> `BASHUNIT_COVERAGE_SHOW_LINE_HITS=true|false` + +Opt-in blocks appended to the console coverage report: a per-function table, the executable +lines never hit (as compressed ranges), and per-line execution counts. All `false` by +default. + ### Coverage engine > `BASHUNIT_COVERAGE_ENGINE=auto|xtrace|trap` @@ -925,9 +1083,9 @@ BASHUNIT_COVERAGE_MIN=80 Thresholds for color-coding the coverage output. Defaults: `50` and `80`. -- Below `THRESHOLD_LOW`: Red -- Between thresholds: Yellow -- Above `THRESHOLD_HIGH`: Green +- At or above `THRESHOLD_HIGH`: green +- At or above `THRESHOLD_LOW`: yellow +- Below `THRESHOLD_LOW`: red ::: code-group ```bash [.env] @@ -938,9 +1096,9 @@ BASHUNIT_COVERAGE_THRESHOLD_HIGH=90 ## Deprecations -Every setting also answers to an **unprefixed** name — `VERBOSE` as well as -`BASHUNIT_VERBOSE`. Those unprefixed aliases predate the `BASHUNIT_` prefix -introduced in 0.15.0 and are deprecated: the names are generic enough that an +Many settings also answer to an **unprefixed** name — `VERBOSE` as well as +`BASHUNIT_VERBOSE`. Those are the ones that predate the `BASHUNIT_` prefix introduced in +0.15.0, and they are deprecated: the names are generic enough that an unrelated tool exporting `COVERAGE=true` or `VERBOSE=true` silently reconfigures bashunit. Always use the prefixed name. @@ -950,6 +1108,10 @@ When a deprecated form is what supplied a value, bashunit says so on stderr: Deprecated: the unprefixed `VERBOSE`. Use `BASHUNIT_VERBOSE` instead. ``` +Settings added after the prefix ship **only** under `BASHUNIT_`, so an unprefixed form for +them is ignored rather than warned about. `--retry`, `--seed`, `--order-by`, `--list`, and +the snapshot and shard settings are all prefix-only. + The warning goes to stderr, so it never corrupts a report on stdout. Silence it with: diff --git a/docs/coverage.md b/docs/coverage.md index 663fc7e1..41dd7433 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -43,10 +43,17 @@ Coverage report written to: coverage/lcov.info bashunit records which lines ran, then classifies and reports them: 1. **Capture**: every executed line's file path and line number is recorded, either from a `DEBUG` trap or from `xtrace` — see [Tracing engine](#tracing-engine) -2. **Filtering**: only files matching your coverage paths (and not excluded) are tracked +2. **Filtering**: a recorded file is tracked only when it matches your coverage paths and no exclude pattern 3. **Aggregation**: after tests complete, hit data is aggregated 4. **Reporting**: each source line is classified as executable or not, and the executable ones are matched against the hits +::: warning Only executed files are reported +A file enters the report the first time one of its lines runs. A source file that no test +touched at all is **absent** from the report rather than shown at 0%, so the percentage is +measured over the files that ran, not over everything under `BASHUNIT_COVERAGE_PATHS`. +Tracked in [#1053](https://github.com/TypedDevs/bashunit/issues/1053). +::: + ::: tip Performance Coverage roughly doubles to quadruples wall-clock time, depending on Bash version and engine. Cost is split fairly evenly between capture and reporting, and the @@ -108,40 +115,23 @@ For most projects following standard naming conventions, you can simply run `bas ### Environment Variables -You can also configure coverage via [environment variables](/configuration) in your `.env` file: +Every coverage flag has a matching setting. The full list, with defaults, lives in +[Configuration > Coverage](/configuration#coverage). The short version: ```bash -# Enable coverage -BASHUNIT_COVERAGE=true - -# Paths to track (comma-separated) -BASHUNIT_COVERAGE_PATHS=src/,lib/ - -# Patterns to exclude (comma-separated) -BASHUNIT_COVERAGE_EXCLUDE=tests/*,vendor/*,*_test.sh - -# LCOV report output path +BASHUNIT_COVERAGE=true # enable tracking +BASHUNIT_COVERAGE_PATHS=src/,lib/ # paths to track BASHUNIT_COVERAGE_REPORT=coverage/lcov.info - -# HTML report output directory (generates line-by-line coverage view) BASHUNIT_COVERAGE_REPORT_HTML=coverage/html - -# Minimum coverage percentage (optional) -BASHUNIT_COVERAGE_MIN=80 - -# Color thresholds for console output -BASHUNIT_COVERAGE_THRESHOLD_LOW=50 # Red below this -BASHUNIT_COVERAGE_THRESHOLD_HIGH=80 # Green above this, yellow between - -# Tracing engine: auto (default), xtrace or trap -BASHUNIT_COVERAGE_ENGINE=auto - -# Optional text-report blocks (off by default, opt-in for verbose runs) -BASHUNIT_COVERAGE_SHOW_FUNCTIONS=true # Print per-function coverage -BASHUNIT_COVERAGE_SHOW_UNCOVERED=true # Print missed line ranges per file -BASHUNIT_COVERAGE_SHOW_LINE_HITS=true # Print per-line execution counts (lineno:count) +BASHUNIT_COVERAGE_MIN=80 # fail below this percentage +BASHUNIT_COVERAGE_ENGINE=auto # auto, xtrace or trap ``` +Three opt-in blocks add detail to the console report: +`BASHUNIT_COVERAGE_SHOW_FUNCTIONS` (per-function table), +`BASHUNIT_COVERAGE_SHOW_UNCOVERED` (missed line ranges), +`BASHUNIT_COVERAGE_SHOW_LINE_HITS` (per-line execution counts). + ### Tracing engine Coverage can capture executed lines two ways, selected with From 88779798b378e794c446d8c275950d9b3caa0407 Mon Sep 17 00:00:00 2001 From: Chemaclass Date: Tue, 11 Aug 2026 14:32:32 +0200 Subject: [PATCH 03/10] fix(cli): list --show-skipped and --show-incomplete in the help text Both flags are accepted by the parser (src/main/test.sh:280,284) and documented in docs/configuration.md, but 'bashunit test --help' never listed them. Also registers BASHUNIT_COVERAGE_SHOW_FUNCTIONS and BASHUNIT_COVERAGE_SHOW_UNCOVERED in src/config/env.sh. They shipped read-only from src/coverage/report_text.sh with a :-false guard and no default, which is why they were absent from .env.example and from every settings list. .env.example now covers all 66 registered settings; 19 were missing, including RETRY, SEED, TEST_TIMEOUT, the shard pair, the snapshot trio, REPORT_TAP, REPORT_JSON, PARALLEL_JOBS and WATCH_INTERVAL. --- .env.example | 29 +++++++++++++++++++++++++++-- CHANGELOG.md | 28 ++++++++++++++-------------- src/config/env.sh | 7 +++++++ src/console/header.sh | 2 ++ 4 files changed, 50 insertions(+), 16 deletions(-) diff --git a/.env.example b/.env.example index 0eb11ea5..2e19f072 100644 --- a/.env.example +++ b/.env.example @@ -35,6 +35,9 @@ BASHUNIT_PROFILE= # Default: false (report slowest tests after BASHUNIT_PROFILE_COUNT= # Default: 10 (how many slowest tests to report) BASHUNIT_NO_COLOR= # Default: false (disable colors) BASHUNIT_NO_DIFF= # Default: false (disable unified diff on multiline assert failures) +BASHUNIT_NO_PROGRESS= # Default: false (suppress real-time progress, final results only) +BASHUNIT_SHOW_OUTPUT_ON_FAILURE= # Default: true (show captured test output on failure) +BASHUNIT_OUTPUT_FORMAT= # Default: empty (tap = TAP version 13 on stdout) #─────────────────────────────────────────────────────────────────────────────── # Test Execution @@ -49,13 +52,22 @@ BASHUNIT_GHA_ANNOTATIONS= # Default: auto (or always, never) BASHUNIT_REPORT_MD= # Default: empty (Markdown summary path) BASHUNIT_CHANGED= # Default: false (run only test files changed since a git ref) BASHUNIT_CHANGED_REF= # Default: empty (--changed ref: origin/HEAD, then HEAD) -BASHUNIT_COVERAGE_DIFF= # Default: empty (restrict coverage to lines changed since this ref) +BASHUNIT_COVERAGE_DIFF= # Default: empty (restrict coverage to lines changed since this ref) BASHUNIT_EXCLUDE_FILTER= # Default: empty (skip tests whose name matches) BASHUNIT_LIST_TESTS= # Default: false (print the tests that would run, run none) BASHUNIT_LIST_FORMAT= # Default: text (--list rendering: text or json) BASHUNIT_STOP_ON_ASSERTION_FAILURE= # Default: true (stop test on first assertion fail) BASHUNIT_STRICT_MODE= # Default: false (enable set -euo pipefail) BASHUNIT_LOGIN_SHELL= # Default: false (source login shell profiles) +BASHUNIT_PARALLEL_JOBS= # Default: 0 (unlimited; N or auto caps concurrency) +BASHUNIT_RANDOM_ORDER= # Default: false (shuffle files and tests) +BASHUNIT_SEED= # Default: empty (pins --random-order for a replay) +BASHUNIT_RETRY= # Default: 0 (re-run a failed test up to N extra times) +BASHUNIT_TEST_TIMEOUT= # Default: 0 (off; fail a test running longer than N seconds) +BASHUNIT_SHARD_INDEX= # Default: empty (run shard i of BASHUNIT_SHARD_TOTAL) +BASHUNIT_SHARD_TOTAL= # Default: empty (how many shards to split the suite into) +BASHUNIT_SKIP_ENV_FILE= # Default: false (skip .env and .bashunitrc) +BASHUNIT_WATCH_INTERVAL= # Default: 3 (seconds between polls in the watch fallback) #─────────────────────────────────────────────────────────────────────────────── # Reports @@ -63,6 +75,15 @@ BASHUNIT_LOGIN_SHELL= # Default: false (source login shell profile BASHUNIT_LOG_JUNIT= # JUnit XML report path (e.g., report.xml) BASHUNIT_LOG_GHA= # GitHub Actions workflow-commands log path (e.g., gha.log) BASHUNIT_REPORT_HTML= # HTML test report path (e.g., report.html) +BASHUNIT_REPORT_TAP= # TAP version 13 report path (e.g., report.tap) +BASHUNIT_REPORT_JSON= # JSON report path (e.g., report.json) + +#─────────────────────────────────────────────────────────────────────────────── +# Snapshots +#─────────────────────────────────────────────────────────────────────────────── +BASHUNIT_SNAPSHOT_UPDATE= # Default: false (rewrite existing snapshots) +BASHUNIT_SNAPSHOT_CREATE= # Default: true (record a missing snapshot instead of failing) +BASHUNIT_SNAPSHOT_REPORT_UNUSED= # Default: false (list snapshots no test resolved) #─────────────────────────────────────────────────────────────────────────────── # Code Coverage @@ -74,7 +95,11 @@ BASHUNIT_COVERAGE_REPORT= # Default: coverage/lcov.info BASHUNIT_COVERAGE_REPORT_HTML= # HTML coverage report directory (e.g., coverage/html) BASHUNIT_COVERAGE_MIN= # Minimum coverage % (fails if below) BASHUNIT_COVERAGE_THRESHOLD_LOW= # Default: 50 (red below this) -BASHUNIT_COVERAGE_THRESHOLD_HIGH= # Default: 80 (green above this) +BASHUNIT_COVERAGE_THRESHOLD_HIGH= # Default: 80 (green at or above this) +BASHUNIT_COVERAGE_ENGINE= # Default: auto (or xtrace, trap) +BASHUNIT_COVERAGE_SHOW_LINE_HITS= # Default: false (print per-line execution counts) +BASHUNIT_COVERAGE_SHOW_FUNCTIONS= # Default: false (print per-function coverage) +BASHUNIT_COVERAGE_SHOW_UNCOVERED= # Default: false (print missed line ranges per file) #─────────────────────────────────────────────────────────────────────────────── # Advanced / Debug diff --git a/CHANGELOG.md b/CHANGELOG.md index b1c1cbaf..d14256a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,28 +3,28 @@ ## Unreleased ### Added -- `--coverage-diff ` restricts the coverage console report to lines changed since a base ref, and `--coverage-min` then gates on that diff percentage (#1032) -- `--report-md ` writes a Markdown run summary: verdict, counts table, failures with their message, plus coverage and slowest tests when those ran. Inside GitHub Actions it is appended to `$GITHUB_STEP_SUMMARY` automatically, so failures render on the job page (#1015) -- `--gha-annotations ` controls GitHub Actions annotations on stdout; `auto` turns them on inside GitHub Actions and stays quiet everywhere else (#1014) -- `--repeat ` runs each selected test n times so flakiness can be hunted before it reaches CI. The test is reported once with the aggregate outcome, a failure names the iteration it happened on, and repeat wraps `--retry` rather than the other way round (#1013) -- Flaky is a first-class outcome: a test that only passed after a retry is counted separately, kept inside the pass total so the exit code is unchanged, and carried into JUnit (``), TAP, JSON, HTML and GitHub Actions along with the first attempt's failure message. `--fail-on-flaky` turns such a run red (#1012) -- `--order-by ` picks the execution order: `defined` (default), `defects` (last run's failures first, whole suite still runs) or `random`. `--random-order` and `--seed` keep working unchanged (#1011) -- `--changed []` runs only the test files git reports as touched since `` (default `origin/HEAD`, then `HEAD`), covering committed, staged, unstaged and untracked changes. Deletions are dropped, a rename selects its new path, and a missing work tree or unresolvable ref fails the run instead of selecting nothing (#1010) +- `--coverage-diff ` limits the coverage console report to lines changed since a base ref; `--coverage-min` then gates on that diff percentage (#1032) +- `assert_command_available ` asserts a command, shell builtin or function resolves through `command -v` (#1027) - `assert_between ` and `assert_not_between` add inclusive numeric-range assertions for integers and decimals (#1026) -- `--list` (alias `--dry-run`) prints the tests a run would execute, without running them; `--list-format json` emits file, function, name, line and tags. Honours every selection flag, including `--shard` and `--random-order --seed` ordering (#1007) +- `--report-md ` writes a Markdown run summary: verdict, counts table, failures with their message, plus coverage and slowest tests when those ran. Inside GitHub Actions it is also appended to `$GITHUB_STEP_SUMMARY` (#1015) +- `--gha-annotations ` controls GitHub Actions annotations on stdout; `auto` enables them only inside GitHub Actions (#1014) +- `--repeat ` runs each selected test n times to hunt flakiness before CI does. One report line with the aggregate outcome, a failure names its iteration, and repeat wraps `--retry` (#1013) +- Flaky is a first-class outcome: a test that only passed after a retry is counted separately, stays inside the pass total so the exit code is unchanged, and is carried into JUnit (``), TAP, JSON, HTML and GitHub Actions with the first attempt's failure message. `--fail-on-flaky` turns such a run red (#1012) +- `--order-by ` picks the execution order: `defined` (default), `defects` (last run's failures first, whole suite still runs) or `random`. `--random-order` and `--seed` are unchanged (#1011) +- `--changed []` runs only the test files git reports as touched since `` (default `origin/HEAD`, then `HEAD`), covering committed, staged, unstaged and untracked changes. Deletions are dropped, a rename selects its new path, and a missing work tree or unresolvable ref fails the run (#1010) - `--exclude-filter ` skips tests by name, the counterpart of `--exclude-tag`. Repeatable, OR'd, and wins over `--filter` (#1009) -- `# @tags a b` above any top-level line applies those tags to every test in the file, unioned with per-function `# @tag` (#1008) - `--tag` accepts expressions: `'a&&b'` (AND) and `'!a'` (NOT), combinable as `'a&&!b'`. Repeated `--tag` flags keep OR semantics, and `--exclude-tag` still wins (#1008) -- `assert_command_available ` asserts that an external command, shell builtin or function resolves through `command -v` (#1027) -- The coverage engine in use is reported by `--verbose`, and an explicit `BASHUNIT_COVERAGE_ENGINE=xtrace` that the running Bash cannot honour now warns instead of being silently ignored (#1005) +- `# @tags a b` above any top-level line tags every test in the file, unioned with per-function `# @tag` (#1008) +- `--list` (alias `--dry-run`) prints the tests a run would execute without running them; `--list-format json` emits file, function, name, line and tags. Honours every selection flag, including `--shard` and `--random-order --seed` ordering (#1007) +- `--verbose` reports the coverage engine in use, and an explicit `BASHUNIT_COVERAGE_ENGINE=xtrace` the running Bash cannot honour now warns instead of being silently ignored (#1005) ### Changed -- The JUnit XML shape changed: one `` per test file (with its own counts, time and timestamp) instead of a single flat suite, `classname` on every ``, `` carrying the first informative line of the real message with `type="AssertionFailed"`, `` with the test's captured output, and aggregate totals on ``. Consumers that group by suite or classname (Jenkins, GitLab, dorny/test-reporter) now get real groupings (#1016) -- Performance: `--coverage` is about 1.6x to 2.3x faster. Executable-line classification no longer forks `grep` per source line, which was roughly half of a coverage run's wall time and affected both engines equally (#1005) +- JUnit XML: one `` per test file with its own counts, time and timestamp instead of a single flat suite, `classname` on every ``, `` carrying the first informative line of the real message with `type="AssertionFailed"`, `` with the test's captured output, and aggregate totals on ``. Consumers that group by suite or classname (Jenkins, GitLab, dorny/test-reporter) now get real groupings (#1016) +- Performance: `--coverage` is about 1.6x to 2.3x faster. Executable-line classification no longer forks `grep` per source line, roughly half of a coverage run's wall time on both engines (#1005) ### Fixed - Build: the standalone binary size budget is 544 KiB, raised from 500 KiB after ordinary feature growth crossed it; the artifact keeps its indentation rather than being minified (#1045) -- `assert_within_delta` rejects malformed numbers such as `1.2.3` or `5-3` as non-numeric instead of leaking a raw `bc` parse error or silently evaluating them as an expression (#1026) +- `assert_within_delta` rejects malformed numbers such as `1.2.3` or `5-3` as non-numeric instead of leaking a raw `bc` parse error or evaluating them as an expression (#1026) - Report formats are no longer empty under `--parallel`. `--report-junit`, `--report-tap`, `--report-json`, `--report-html` and `--log-junit` all recorded zero tests, because the rows were collected inside the per-test worker and nothing rebuilt them in the parent (#1004) ## [0.45.0](https://github.com/TypedDevs/bashunit/compare/0.44.0...0.45.0) - 2026-08-09 diff --git a/src/config/env.sh b/src/config/env.sh index c56842f2..1d4dafa6 100644 --- a/src/config/env.sh +++ b/src/config/env.sh @@ -177,6 +177,9 @@ _BASHUNIT_DEFAULT_COVERAGE_THRESHOLD_LOW="50" _BASHUNIT_DEFAULT_COVERAGE_THRESHOLD_HIGH="80" # Per-line execution counts in the text coverage report (#856) _BASHUNIT_DEFAULT_COVERAGE_SHOW_LINE_HITS="false" +# Opt-in text-report blocks, read by src/coverage/report_text.sh +_BASHUNIT_DEFAULT_COVERAGE_SHOW_FUNCTIONS="false" +_BASHUNIT_DEFAULT_COVERAGE_SHOW_UNCOVERED="false" # Tracing engine: auto|xtrace|trap. auto takes the xtrace fast path wherever # BASH_XTRACEFD exists (Bash 4.1+) and the DEBUG trap below it (ADR-009, #860) _BASHUNIT_DEFAULT_COVERAGE_ENGINE="auto" @@ -212,6 +215,10 @@ BASHUNIT_WATCH_INTERVAL=$(bashunit::env::positive_int_or_default \ # no-op consolidation, whereas adding the alias would widen the public API. # bashunit::coverage keeps its :- guard for callers that unset it. : "${BASHUNIT_COVERAGE_SHOW_LINE_HITS:=$_BASHUNIT_DEFAULT_COVERAGE_SHOW_LINE_HITS}" +# Same reasoning for the other two text-report blocks, which shipped read-only +# from src/coverage/report_text.sh and had no default registered here. +: "${BASHUNIT_COVERAGE_SHOW_FUNCTIONS:=$_BASHUNIT_DEFAULT_COVERAGE_SHOW_FUNCTIONS}" +: "${BASHUNIT_COVERAGE_SHOW_UNCOVERED:=$_BASHUNIT_DEFAULT_COVERAGE_SHOW_UNCOVERED}" # No bare COVERAGE_ENGINE alias: the unprefixed forms are deprecated, so a new # setting only ever ships under the BASHUNIT_ prefix. : "${BASHUNIT_COVERAGE_ENGINE:=$_BASHUNIT_DEFAULT_COVERAGE_ENGINE}" diff --git a/src/console/header.sh b/src/console/header.sh index 13f76937..9c46c996 100644 --- a/src/console/header.sh +++ b/src/console/header.sh @@ -157,6 +157,8 @@ Options: --debug [file] Enable shell debug mode --no-output Suppress all output --failures-only Only show failures (suppress passed/skipped/incomplete) + --show-skipped Show the skipped tests summary at the end + --show-incomplete Show the incomplete tests summary at the end --fail-on-risky Treat risky tests (no assertions) as failures --fail-on-flaky Treat flaky tests (passed only after a retry) as failures --profile Report the slowest tests (count: BASHUNIT_PROFILE_COUNT, default 10) From a848828ebaacedcf62f57db50ea09bf0ecb7ae01 Mon Sep 17 00:00:00 2001 From: Chemaclass Date: Tue, 11 Aug 2026 14:35:11 +0200 Subject: [PATCH 04/10] docs(command-line): refresh the output blocks and close the reference gaps Every example output in this commit was reproduced against the real binary. Corrections: the JSON schema was missing the flaky summary key and the per-test retries field; the page claimed per-test rows only come from a sequential run, which #1004 fixed; the --retry example predated the flaky counter; --profile prints Time taken before the slowest list, not after; the GitHub Actions annotation sample was not percent-encoded although the prose next to it says it is; the Markdown failure fence repeats the test name and location; init also writes .github/workflows/tests.yml and .env; two output blocks still showed 0.34.1; --coverage-exclude was missing *Test.sh. Additions: a ## assert section for the subcommand that had none, -h/--help in the test table, --no-color and -h in the bench table, --repeat and the three enum checks in the invalid-values list, and the watch constraint that only -f/--filter forwards its value. Deduplication: the eight coverage flags were tabulated twice with disagreeing defaults, so the Test Options table now points at the Coverage section, and the Diff coverage explainer moved to docs/coverage.md, which the page already names as the owner of coverage detail. --- docs/command-line.md | 143 ++++++++++++++++++++----------------------- docs/coverage.md | 61 +++++++++++++++++- 2 files changed, 128 insertions(+), 76 deletions(-) diff --git a/docs/command-line.md b/docs/command-line.md index 972e7f69..0c787287 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -58,7 +58,7 @@ bashunit test tests/ --parallel --simple | Option | Description | |--------------------------------|--------------------------------------------------| -| `-a, --assert ` | Run a standalone assert function | +| `-a, --assert ` | Run a standalone assert function (deprecated: use [`bashunit assert`](#assert)) | | `-e, --env, --boot ` | Load custom env/bootstrap file (supports args) | | `-f, --filter ` | Only run tests matching name | | `--exclude-filter ` | Skip tests whose name matches (repeatable) | @@ -110,19 +110,18 @@ bashunit test tests/ --parallel --simple | `--skip-env-file` | Skip `.env` loading, use shell environment only | | `-l, --login` | Run tests in login shell context | | `--no-color` | Disable colored output | -| `--coverage` | Enable code coverage tracking | -| `--coverage-paths ` | Paths to track (default: auto-discover) | -| `--coverage-exclude ` | Exclusion patterns | -| `--coverage-report [file]` | LCOV output path (default: `coverage/lcov.info`) | -| `--coverage-report-html [dir]` | Generate HTML report (default: `coverage/html`) | -| `--coverage-min ` | Minimum coverage threshold | -| `--coverage-diff ` | Report coverage only for lines changed since ref | -| `--no-coverage-report` | Console output only, no LCOV file | +| `-h, --help` | Show the test help | +| `--coverage*` | Eight coverage flags, see [Coverage](#coverage) | ### Standalone Assert > `bashunit test -a|--assert function "arg1" "arg2"` +::: warning Deprecated +Use the [`assert` subcommand](#assert) instead. This form still works and prints a +deprecation notice on stderr. +::: + Run a core assert function standalone without a test context. ::: code-group @@ -130,7 +129,8 @@ Run a core assert function standalone without a test context. bashunit test --assert equals "foo" "bar" ``` ```[Output] -✗ Failed: Main::exec assert +Deprecated: `bashunit test --assert`. Use `bashunit assert` instead. +✗ Failed: assert equals Expected 'foo' but got 'bar' ``` @@ -485,8 +485,10 @@ developer actually looks at first: `tests/math_test.sh:42` ``` -Expected '4' -but got '5' +✗ Failed: Sums two numbers + Expected '4' + but got '5' + at tests/math_test.sh:42 ``` ``` @@ -515,7 +517,7 @@ Inside GitHub Actions, bashunit annotates failing tests on the pull request by itself. No flag, no configuration: ``` -::error file=tests/math_test.sh,line=42,title=Sums::Expected '4' but got '5' +::error file=tests/math_test.sh,line=42,title=Sums two numbers::✗ Failed: Sums two numbers%0A Expected '4'%0A but got '5'%0A at tests/math_test.sh:42 ``` GitHub parses workflow commands from the **job log**, so the annotations go to @@ -548,15 +550,15 @@ The `--report-json` flag writes machine-readable results for scripts, dashboards ```json { - "summary": { "total": 3, "passed": 2, "failed": 1, "skipped": 0, "incomplete": 0, "duration_ms": 42 }, + "summary": { "total": 3, "passed": 2, "failed": 1, "skipped": 0, "incomplete": 0, "flaky": 0, "duration_ms": 42 }, "tests": [ - { "file": "tests/math_test.sh", "name": "it adds", "status": "passed", "duration_ms": 5, "message": "" }, - { "file": "tests/math_test.sh", "name": "it divides", "status": "failed", "duration_ms": 3, "message": "Expected 2 but got 3" } + { "file": "tests/math_test.sh", "name": "it adds", "status": "passed", "duration_ms": 5, "retries": 0, "message": "" }, + { "file": "tests/math_test.sh", "name": "it divides", "status": "failed", "duration_ms": 3, "retries": 0, "message": "Expected 2 but got 3" } ] } ``` -`status` is one of `passed`, `failed`, `skipped`, `incomplete` (`snapshot` and `risky` are also emitted per test and counted as passed in the summary). Like the other file reporters, per-test rows come from a sequential run; under `--parallel` the file is still valid JSON. +`status` is one of `passed`, `failed`, `skipped`, `incomplete`, `flaky` (`snapshot` and `risky` are also emitted per test and counted as passed in the summary). Per-test rows are complete in both modes; under `--parallel` the row order follows completion order rather than definition order. ### Show Output on Failure @@ -606,12 +608,12 @@ Tests: 10 passed, 10 total Assertions: 25 passed, 25 total All tests passed +Time taken: 1.60s Slowest tests: 1.20s test_slow_database_query (tests/integration_test.sh) 340ms test_http_client_timeout (tests/http_test.sh) 12ms test_parse_config (tests/unit/config_test.sh) -Time taken: 1.60s ``` ```bash [Custom count] BASHUNIT_PROFILE_COUNT=3 bashunit test tests/ --profile @@ -667,8 +669,11 @@ bashunit test tests/ --retry 2 ```[Output] ✓ Passed: A flaky test (retry 1/2) -Tests: 1 passed, 1 total +Tests: 1 passed, 1 flaky, 1 total ``` + +A test that recovered on retry is counted as [flaky](#flaky-tests) as well as passed, so +the exit code stays `0` unless `--fail-on-flaky` is set. ::: It can also be set via the `BASHUNIT_RETRY` environment variable (see @@ -1111,7 +1116,7 @@ This is useful for: bashunit test tests/ --no-progress ``` ```[Output] -bashunit - 0.34.1 | Tests: 10 +bashunit | Tests: 10 Tests: 10 passed, 10 total Assertions: 25 passed, 25 total @@ -1208,7 +1213,7 @@ bashunit test tests/ --coverage --coverage-paths src/,lib/ --coverage-min 80 |---------------------------------|-----------------------------------------------------------------------------| | `--coverage` | Enable coverage tracking | | `--coverage-paths ` | Comma-separated paths to track (default: auto-discover from test files) | -| `--coverage-exclude ` | Comma-separated patterns to exclude (default: `tests/*,vendor/*,*_test.sh`) | +| `--coverage-exclude ` | Comma-separated patterns to exclude (default: `tests/*,vendor/*,*_test.sh,*Test.sh`) | | `--coverage-report [file]` | LCOV output file path (default: `coverage/lcov.info`) | | `--coverage-report-html [dir]` | Generate HTML report (default: `coverage/html`) | | `--coverage-min ` | Minimum coverage percentage; fails if below | @@ -1223,60 +1228,31 @@ Coverage works with parallel execution (`-p`). Each worker tracks coverage indep > `bashunit test --coverage --coverage-diff ` -Answers the question a pull request actually asks — *are the lines I touched -covered?* — instead of reporting a whole-file percentage that moves for reasons -unrelated to the change under review. +Restrict the console report to the lines changed since a base ref, so a pull request is +judged on the code it touched. See [Coverage > Diff coverage](/coverage#diff-coverage). -```bash -bashunit test tests/ --coverage --coverage-diff main -``` - -``` -Diff Coverage (vs main) ---------------- -src/parser.sh 7/ 9 lines ( 77%) ---------------- -Total: 7/9 (77%) -``` +## assert -Only lines **added or modified** since the ref are counted, from three sources -merged together: commits since the merge base, staged and unstaged edits, and -untracked files (counted in full). A pure deletion contributes nothing — there -is no line left to hold an opinion about — and changed lines that are not -executable (comments, `fi`, blank) are ignored, so a comment-only commit is not -penalised. +> `bashunit assert [args...]` +> +> `bashunit assert "" [ ...]` -The base ref is **required**. It is not defaulted, because an optional value -would make `--coverage-diff tests/` swallow the path as a ref. +Run assertions without creating a test file. The function name works with or without the +`assert_` prefix. -**With `--coverage-min`, the gate follows the report:** the threshold applies to -the diff percentage, so a change that fully covers itself passes even inside a -poorly covered file. - -```bash -bashunit test tests/ --coverage --coverage-diff origin/main --coverage-min 90 +::: code-group +```bash [Single assertion] +bashunit assert equals "foo" "foo" +bashunit assert exit_code 0 "echo 'success'" ``` - -A change with no executable lines scores **100%**, not 0% — otherwise a -docs-only commit would fail the gate. - -`--coverage-diff` restricts the **console report only**. LCOV and HTML stay -whole-file, because their consumers (`genhtml`, Codecov) do their own diffing -and expect complete records. - -::: warning Shallow clones -This needs `git` and a ref that resolves locally. CI checkouts are often shallow -and have no base ref, which would otherwise report "no changed lines" and pass a -threshold while measuring nothing — so an unresolvable ref is a hard error -instead. Fetch it first: - -```yaml -- uses: actions/checkout@v4 - with: - fetch-depth: 0 +```bash [Several assertions on one command] +bashunit assert "./my_script.sh" exit_code "0" contains "success" not_contains "error" ``` ::: +See [Standalone](/standalone) for the full story. `bashunit test --assert` is the +deprecated form of the first mode. + ## bench > `bashunit bench [path] [options]` @@ -1307,6 +1283,8 @@ bashunit bench --filter "parse" | `-vvv, --verbose` | Show execution details | | `--skip-env-file` | Skip `.env` loading, use shell environment only | | `-l, --login` | Run in login shell context | +| `--no-color` | Disable colored output (honors `NO_COLOR`) | +| `-h, --help` | Show the bench help | ## watch @@ -1314,10 +1292,13 @@ bashunit bench --filter "parse" Dedicated watch subcommand that uses **OS file-event notifications** (no polling) to re-run tests as soon as a `.sh` file changes. Any option accepted -by `bashunit test` is also accepted here. +by `bashunit test` is also accepted here, **but put the path first**: apart from +`-f/--filter`, an option's value is otherwise taken as the watch path, so +`bashunit watch --tag slow` watches a directory named `slow`. Write +`bashunit watch tests/ --tag slow`. When neither `inotifywait` nor `fswatch` is installed, it no longer fails: -it falls back to a **pure-shell polling loop** and prints a one-line notice. +it falls back to a **pure-shell polling loop** and prints a short notice. Polling checks every `BASHUNIT_WATCH_INTERVAL` seconds (default `2`) using `find -newer`, so it detects created and modified `.sh` files; deleted files are not detected on the fallback path. Install one of the tools above for @@ -1344,8 +1325,9 @@ bashunit watch tests/ --simple - **macOS:** `fswatch` (`brew install fswatch`) Without either tool, bashunit degrades to polling (see above) instead of -failing. The portable [`-w/--watch`](#watch-mode) flag on `bashunit test` -also uses polling. +failing. The portable [`-w/--watch`](#watch-mode) flag on `bashunit test` also polls, but +on a fixed 1-second loop: `BASHUNIT_WATCH_INTERVAL` applies to this subcommand's fallback +only. ::: ## doc @@ -1405,13 +1387,21 @@ bashunit init bashunit init spec ``` ```[Output] +> Created tests/bootstrap.sh +> Created tests/example_test.sh +> Created .github/workflows/tests.yml > bashunit initialized in tests ``` ::: Creates: -- `bootstrap.sh` - Setup file for test configuration -- `example_test.sh` - Sample test file to get started +- `tests/bootstrap.sh` - Setup file for test configuration +- `tests/example_test.sh` - Sample test file to get started +- `.github/workflows/tests.yml` - CI workflow using the official action +- `.env` with `BASHUNIT_BOOTSTRAP=tests/bootstrap.sh` + +An existing `BASHUNIT_BOOTSTRAP=` line in `.env` is commented out first, so the new value +takes effect. ## learn @@ -1463,7 +1453,7 @@ bashunit upgrade ``` ```[Output] > Upgrading bashunit to latest version -> bashunit upgraded successfully to latest version 0.34.1 +> bashunit upgraded successfully to latest version ``` ::: @@ -1549,7 +1539,9 @@ Without this, `--parralel` ran the suite **sequentially** and still exited `0`, ### Invalid values `--jobs`, `--retry`, `--test-timeout`, `--coverage-min` and `--seed` require a -non-negative integer; `--output` accepts only `tap`; `--shard` requires `/`: +non-negative integer; `--repeat` requires an integer of at least `1`; `--output` accepts +only `tap`; `--shard` requires `/`; and `--gha-annotations`, `--order-by` and +`--list-format` accept only their listed modes: ```bash bashunit --jobs abc tests/ @@ -1592,3 +1584,4 @@ risky also exits `0`. Add [`--fail-on-risky`](#test-options) or read the counts +- [Standalone](/standalone) — run assertions without a test file diff --git a/docs/coverage.md b/docs/coverage.md index 41dd7433..42435cf6 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -18,7 +18,7 @@ bashunit tests/ --coverage bashunit tests/ --coverage-paths src/ ``` ```bash [Output] -bashunit - 0.37.0 | Tests: 5 +bashunit | Tests: 5 ..... Tests: 5 passed, 5 total @@ -92,6 +92,7 @@ Warning: coverage engine 'xtrace' needs Bash 4.1+ (running 3.2); using 'trap'. | `--coverage-report ` | LCOV report output path (default: `coverage/lcov.info`) | | `--coverage-report-html [dir]` | Generate HTML report (default: `coverage/html`) | | `--coverage-min ` | Minimum coverage threshold (fails if below) | +| `--coverage-diff ` | Report only the lines changed since ``, see [Diff coverage](#diff-coverage) | | `--no-coverage-report` | Disable LCOV file generation (console only) | ::: tip Auto-enable @@ -550,6 +551,64 @@ Note the workflow excludes the engine's own meta-tests (`tests/unit/coverage_*_t from the measured run: executing them under `--coverage` double-instruments `src/coverage.sh` and corrupts their assertions. +## Diff coverage + +> `bashunit test --coverage --coverage-diff ` + +Answers the question a pull request actually asks — *are the lines I touched +covered?* — instead of reporting a whole-file percentage that moves for reasons +unrelated to the change under review. + +```bash +bashunit test tests/ --coverage --coverage-diff main +``` + +``` +Diff Coverage (vs main) +--------------- +src/parser.sh 7/ 9 lines ( 77%) +--------------- +Total: 7/9 (77%) +``` + +Only lines **added or modified** since the ref are counted, from three sources +merged together: commits since the merge base, staged and unstaged edits, and +untracked files (counted in full). A pure deletion contributes nothing — there +is no line left to hold an opinion about — and changed lines that are not +executable (comments, `fi`, blank) are ignored, so a comment-only commit is not +penalised. + +The base ref is **required**. It is not defaulted, because an optional value +would make `--coverage-diff tests/` swallow the path as a ref. + +**With `--coverage-min`, the gate follows the report:** the threshold applies to +the diff percentage, so a change that fully covers itself passes even inside a +poorly covered file. + +```bash +bashunit test tests/ --coverage --coverage-diff origin/main --coverage-min 90 +``` + +A change with no executable lines scores **100%**, not 0% — otherwise a +docs-only commit would fail the gate. + +`--coverage-diff` restricts the **console report only**. LCOV and HTML stay +whole-file, because their consumers (`genhtml`, Codecov) do their own diffing +and expect complete records. + +::: warning Shallow clones +This needs `git` and a ref that resolves locally. CI checkouts are often shallow +and have no base ref, which would otherwise report "no changed lines" and pass a +threshold while measuring nothing — so an unresolvable ref is a hard error +instead. Fetch it first: + +```yaml +- uses: actions/checkout@v4 + with: + fetch-depth: 0 +``` +::: + ## Related - [Command-Line](/command-line) — full reference for CLI flags and options From 1258a8c2e277077fe324902a82029f477d168116 Mon Sep 17 00:00:00 2001 From: Chemaclass Date: Tue, 11 Aug 2026 14:42:13 +0200 Subject: [PATCH 05/10] fix(cli): default --coverage-report and split JSON tags on commas Two defects the docs audit surfaced while checking documented behaviour. --coverage-report is documented with an optional value and behaves like --coverage-report-html, but it read $2 unconditionally: omitting the value aborted the run with '$2: unbound variable' before any test ran, and a following flag was consumed as the filename. It now falls back to coverage/lcov.info, and the docs state that a path must be written before the flag because an optional value cannot be told apart from a test path. --list-format json split the tag list on whitespace while every other consumer splits it on commas (src/helper/tags.sh:137). A test with two tags rendered as the single element "slow,fileTag", so the jq recipe documented on docs/command-line.md matched nothing, and a tag containing spaces was split into one element per word. --- docs/command-line.md | 5 ++++ docs/coverage.md | 2 +- src/main/test.sh | 14 +++++++-- src/runner/list.sh | 12 ++++++++ .../bashunit_invalid_option_value_test.sh | 29 ++++++++++++++++++ tests/acceptance/bashunit_list_test.sh | 30 +++++++++++++++++++ tests/acceptance/fixtures/list/multitag.sh | 14 +++++++++ 7 files changed, 103 insertions(+), 3 deletions(-) create mode 100644 tests/acceptance/fixtures/list/multitag.sh diff --git a/docs/command-line.md b/docs/command-line.md index 0c787287..bbaf9ccb 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -1220,6 +1220,11 @@ bashunit test tests/ --coverage --coverage-paths src/,lib/ --coverage-min 80 | `--coverage-diff ` | Report only the lines changed since `` | | `--no-coverage-report` | Show console report only, don't generate LCOV file | +Both `--coverage-report` and `--coverage-report-html` take an optional value and fall back +to their default path when the next argument is another flag or absent. A value cannot be +told apart from a test path, so write the path before them: `bashunit tests/ --coverage-report`. + + ::: tip Coverage works with parallel execution (`-p`). Each worker tracks coverage independently, and results are aggregated before reporting. ::: diff --git a/docs/coverage.md b/docs/coverage.md index 42435cf6..5b41012f 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -89,7 +89,7 @@ Warning: coverage engine 'xtrace' needs Bash 4.1+ (running 3.2); using 'trap'. | `--coverage` | Enable code coverage tracking | | `--coverage-paths ` | Comma-separated paths to track (default: auto-discover from test files) | | `--coverage-exclude ` | Comma-separated exclusion patterns | -| `--coverage-report ` | LCOV report output path (default: `coverage/lcov.info`) | +| `--coverage-report [file]` | LCOV report output path (default: `coverage/lcov.info`) | | `--coverage-report-html [dir]` | Generate HTML report (default: `coverage/html`) | | `--coverage-min ` | Minimum coverage threshold (fails if below) | | `--coverage-diff ` | Report only the lines changed since ``, see [Diff coverage](#diff-coverage) | diff --git a/src/main/test.sh b/src/main/test.sh index 05abe74a..2c96dcfe 100644 --- a/src/main/test.sh +++ b/src/main/test.sh @@ -349,10 +349,20 @@ function bashunit::main::cmd_test() { shift ;; --coverage-report) + # The value is optional, matching --coverage-report-html below and the + # `[file]` notation the docs use. Reading "$2" unconditionally aborted the + # run with `$2: unbound variable`, or took a following flag as the path. # shellcheck disable=SC2034 - BASHUNIT_COVERAGE_REPORT="$2" + case "${2:-}" in + '' | -*) + BASHUNIT_COVERAGE_REPORT="$_BASHUNIT_DEFAULT_COVERAGE_REPORT" + ;; + *) + BASHUNIT_COVERAGE_REPORT="$2" + shift + ;; + esac _bashunit_coverage_opt_set=true - shift ;; --coverage-min) # shellcheck disable=SC2034 diff --git a/src/runner/list.sh b/src/runner/list.sh index 9d69ed8f..dd900874 100644 --- a/src/runner/list.sh +++ b/src/runner/list.sh @@ -55,7 +55,19 @@ function bashunit::runner::list_functions() { tags="$_BASHUNIT_TAGS_OUT" tags_json="" + # Tags are comma-separated because a tag may contain spaces (`# @tag needs a + # db`), the same split every other consumer uses (src/helper/tags.sh:137). + # IFS is restored right after the split: the helpers called below run under + # the caller's dynamic scope and must not see a comma-only IFS. + local old_ifs="$IFS" + IFS=',' + local -a tag_list=() for tag in $tags; do + tag_list[${#tag_list[@]}]="$tag" + done + IFS="$old_ifs" + + for tag in ${tag_list[@]+"${tag_list[@]}"}; do [ -z "$tag" ] && continue [ -n "$tags_json" ] && tags_json="$tags_json," tags_json="$tags_json\"$(bashunit::reports::__json_escape "$tag")\"" diff --git a/tests/acceptance/bashunit_invalid_option_value_test.sh b/tests/acceptance/bashunit_invalid_option_value_test.sh index e1c4cd46..501ade58 100644 --- a/tests/acceptance/bashunit_invalid_option_value_test.sh +++ b/tests/acceptance/bashunit_invalid_option_value_test.sh @@ -126,3 +126,32 @@ function test_bashunit_still_accepts_tap_output() { assert_successful_code "" "" "$ec" assert_contains "TAP version 13" "$output" } + +# `--coverage-report` is documented with the page's optional notation +# (`[file]`), and --coverage-report-html already defaults its value. Omitting the +# value used to abort the run with `$2: unbound variable` before any test ran, +# and a following flag was consumed as the filename. +function test_coverage_report_without_a_value_uses_the_default_path() { + local dir output + dir="$(bashunit::temp_dir)" + + # The path goes first: an optional value cannot be told apart from a test path, + # which is why --coverage-diff requires its ref (src/main/test.sh). + output=$(cd "$dir" && "$OLDPWD/bashunit" --env "$OLDPWD/$TEST_ENV_FILE" \ + "$OLDPWD/$TEST_FILE" --coverage --coverage-report 2>&1) || true + + assert_not_contains "unbound variable" "$output" + assert_file_exists "$dir/coverage/lcov.info" +} + +function test_coverage_report_does_not_swallow_the_next_flag_as_its_value() { + local dir output + dir="$(bashunit::temp_dir)" + + output=$(cd "$dir" && "$OLDPWD/bashunit" --env "$OLDPWD/$TEST_ENV_FILE" \ + --coverage --coverage-report --no-color "$OLDPWD/$TEST_FILE" 2>&1) || true + + assert_not_contains "unbound variable" "$output" + assert_not_contains "No such file or directory" "$output" + assert_file_exists "$dir/coverage/lcov.info" +} diff --git a/tests/acceptance/bashunit_list_test.sh b/tests/acceptance/bashunit_list_test.sh index f999d613..baf2eccc 100644 --- a/tests/acceptance/bashunit_list_test.sh +++ b/tests/acceptance/bashunit_list_test.sh @@ -14,6 +14,7 @@ ALPHA="$FIXTURES_PATH/alpha.sh" BETA="$FIXTURES_PATH/beta.sh" TAGGED="$FIXTURES_PATH/tagged.sh" ORDER="$FIXTURES_PATH/order.sh" +MULTITAG="$FIXTURES_PATH/multitag.sh" function test_list_prints_every_test_as_file_and_function() { local output @@ -177,6 +178,35 @@ function test_list_format_json_reports_tags() { assert_same "slow" "$(printf '%s' "$output" | jq -r '.tests[0].tags[0]')" } +# The JSON emitter used to split the tag list on whitespace while every other +# consumer splits it on commas, so a test with two tags rendered as the single +# element "slow,fileTag" and the documented jq recipe matched nothing. +function test_list_format_json_reports_each_tag_as_its_own_element() { + if ! command -v jq >/dev/null 2>&1; then + bashunit::skip "jq is required to validate the JSON shape" && return + fi + + local output + output="$(./bashunit --list --list-format json "$MULTITAG" 2>/dev/null)" + + assert_same "2" "$(printf '%s' "$output" | jq -r '.tests[0].tags | length')" + assert_same "slow" "$(printf '%s' "$output" | jq -r '.tests[0].tags[0]')" + assert_same "fileTag" "$(printf '%s' "$output" | jq -r '.tests[0].tags[1]')" +} + +# A tag may contain spaces (`# @tag needs a db`), so it must survive as one +# element rather than being split into three. +function test_list_format_json_keeps_a_tag_containing_spaces_intact() { + if ! command -v jq >/dev/null 2>&1; then + bashunit::skip "jq is required to validate the JSON shape" && return + fi + + local output + output="$(./bashunit --list --list-format json "$MULTITAG" 2>/dev/null)" + + assert_same "needs a db" "$(printf '%s' "$output" | jq -r '.tests[1].tags[0]')" +} + function test_list_format_json_is_valid_for_an_empty_selection() { if ! command -v jq >/dev/null 2>&1; then bashunit::skip "jq is required to validate the JSON shape" && return diff --git a/tests/acceptance/fixtures/list/multitag.sh b/tests/acceptance/fixtures/list/multitag.sh new file mode 100644 index 00000000..02797bbb --- /dev/null +++ b/tests/acceptance/fixtures/list/multitag.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash + +# A tag may contain spaces, so tags are joined and split on commas only. +# @tags fileTag + +# @tag slow +function test_multitag_slow() { + assert_true true +} + +# @tag needs a db +function test_multitag_spaced_tag() { + assert_true true +} From e8d576eb5777072b553fa14b0acbbe6163d76e56 Mon Sep 17 00:00:00 2001 From: Chemaclass Date: Tue, 11 Aug 2026 14:44:03 +0200 Subject: [PATCH 06/10] docs(custom-asserts): correct the API story and the provider gotchas Best practice 3 told readers to call state::add_assertions_passed, which does not exist under that name and aborts the test with 'command not found'. Every bashunit::assert_that example ended the custom assertion with the call itself, so its documented 'return 1' escaped the test function and one failure was reported twice, as a failure plus a spurious Error. The examples now end with return 0 and the API entry says why. Verified by running them. Also: the 'Fails with' comment quoted the humanised test name instead of the message, bashunit::fail was used in examples but missing from the API reference, the bashunit doc output block showed one docstring line where the whole comment run is printed, the --custom filter example dropped --boot and therefore printed 'No custom assertions found', the guard box did not mention -R/--run-all, the assert_once parameter table existed twice, and two examples used [[ ]] against house style, as did the fixture they mirror. data-providers gains the four gotchas it never documented: the two-line annotation proximity rule, a provider with no rows silently making its test disappear while the suite stays green, one concurrent job per row under --parallel, and providers living in a bootstrap. Plus how a failing row is labelled and how rows behave under --repeat, --retry, --tag, --filter and --list. --- docs/custom-asserts.md | 39 +++++++++++++----- docs/data-providers.md | 64 ++++++++++++++++++++++++++++++ tests/functional/custom_asserts.sh | 4 +- 3 files changed, 95 insertions(+), 12 deletions(-) diff --git a/docs/custom-asserts.md b/docs/custom-asserts.md index 62a24b10..07f6bb65 100644 --- a/docs/custom-asserts.md +++ b/docs/custom-asserts.md @@ -12,6 +12,10 @@ Check the internal functional tests: `tests/functional/custom_asserts_test.sh` ( ::: info Assertion behavior When using the bashunit facade, assertions automatically respect the guard behavior: if a previous assertion in the same test already failed, subsequent assertions are skipped. This matches popular testing libraries default behavior. + +This is the default. `-R`/`--run-all` (`BASHUNIT_STOP_ON_ASSERTION_FAILURE=false`) disables +the guard, so every assertion inside your custom assertion runs and counts even after an +earlier failure. ::: ::: info Test name detection @@ -33,6 +37,10 @@ Runs `cmd` and marks the assertion passed or failed accordingly, in a single cal Returns `0` when the command succeeds and `1` when it fails, so it can be chained. +Do not leave it as the **last** statement of your custom assertion: the non-zero status +escapes the test function and the runner reports an extra `✗ Error` on top of the failure. +End the function with `return 0`. + The command is invoked directly, without `eval`, so arguments keep their word boundaries and nothing is re-parsed by the shell. @@ -65,6 +73,12 @@ Marks the current assertion as failed and prints a failure message. Marks the current assertion as passed. Call this when your custom assertion succeeds. +### fail +> `bashunit::fail ` + +Marks the current assertion as failed and prints `Message: ''` instead of the +Expected/but-got block. Use it when there is no meaningful expected value. + ## Examples ### One-call assertion @@ -75,6 +89,7 @@ so the two counters cannot drift apart: ```bash function assert_positive_number() { bashunit::assert_that "positive number" "$1" test "$1" -gt 0 + return 0 } function test_value_is_positive() { @@ -91,10 +106,12 @@ Any command works as the verdict, not only `test`: ```bash function assert_valid_json() { bashunit::assert_that "valid JSON" "$1" jq -e . <<< "$1" + return 0 } function assert_file_is_executable() { bashunit::assert_that "an executable file" "$1" test -x "$1" + return 0 } ``` @@ -123,7 +140,7 @@ function assert_http_success() { function assert_foo() { local actual="$1" - if [[ "foo" != "$actual" ]]; then + if [ "foo" != "$actual" ]; then bashunit::assertion_failed "foo" "$actual" return fi @@ -136,7 +153,7 @@ function test_value_is_foo() { } function test_value_is_not_foo() { - assert_foo "bar" # Fails with: "Failed: Value is not foo" + assert_foo "bar" # Fails with: "Expected 'foo' but got 'bar'" } ``` @@ -204,10 +221,7 @@ Assertions: 1 passed, 1 failed, 2 total The failure now reads `Expected 'a 2xx status' but got '500'`, labelled with the test that called it. -| Parameter | Description | -|-----------|-------------| -| `label` | What the assertion expects, shown in the failure block. Omit it to keep reporting the innermost failure message | -| `actual` | The actual value shown against that label | +Parameters: see [assert_once](#assert-once). Notes: @@ -228,7 +242,7 @@ Notes: function assert_positive_number() { local actual="$1" - if [[ "$actual" -le 0 ]]; then + if [ "$actual" -le 0 ]; then bashunit::assertion_failed "positive number" "$actual" "got" return fi @@ -310,15 +324,20 @@ assertions, it appends them too, rendering the comment block above each one: ```bash ./bashunit doc --boot tests/bootstrap.sh # built-ins, then a "Custom assertions" section ./bashunit doc --custom --boot tests/bootstrap.sh # only your own -./bashunit doc --custom http # ...narrowed by a filter +./bashunit doc --custom --boot tests/bootstrap.sh http # ...narrowed by a filter ``` ``` ## assert_http_success -------------- Asserts that the status code is a 2xx. +Arguments: $1 - the status code ``` +The whole comment run above the function is printed, so a two-line docstring renders as two +lines. A function with no comment block prints its heading and the divider with nothing +under them. + With `BASHUNIT_BOOTSTRAP` set, the `--boot` flag can be omitted. A bootstrap is required either way: it is the only point at which your assertions are guaranteed loaded, which is another reason to prefer it over sourcing from @@ -340,11 +359,11 @@ function assert_http_success() { ## Best practices -1. **Prefer `bashunit::assert_that`**: one call marks the assertion passed or failed, so you cannot forget the `return` after a failure (which would bump both counters) or forget `bashunit::assertion_passed` (which would leave the test with zero assertions, reported as risky). +1. **Prefer `bashunit::assert_that`**: one call marks the assertion passed or failed, so you cannot forget the `return` after a failure (which would bump both counters) or forget `bashunit::assertion_passed` (which would leave the test with zero assertions, reported as risky). End the function with `return 0` so its failure status does not escape the test. 2. **Always return after failure**: when writing the long form by hand, call `return` after `bashunit::assertion_failed` or `bashunit::fail` to stop execution of your custom assertion. -3. **Always mark success**: Call `bashunit::assertion_passed` or `state::add_assertions_passed` when your assertion succeeds. +3. **Always mark success**: call `bashunit::assertion_passed` when your assertion succeeds. 4. **Use descriptive names**: Name your custom assertions clearly, e.g., `assert_valid_email`, `assert_file_contains_header`. diff --git a/docs/data-providers.md b/docs/data-providers.md index 919968af..96319b2f 100644 --- a/docs/data-providers.md +++ b/docs/data-providers.md @@ -25,12 +25,49 @@ function test_my_test_case() { ``` ::: +The annotation must sit within **two lines** of the function definition. A `# @tag` line or +a one-line docstring in between is fine; anything longer and bashunit stops seeing the +annotation, and the test runs once with no arguments instead of reporting an error. + ## Implementing a data provider A data provider function contains one or more `bashunit::data_set` lines. Each `bashunit::data_set` results in a separate run of the test function with the individual `bashunit::data_set` arguments being passed to it as positional arguments (`$1`, `$2`, ...). Each run is treated as a separate test, so it can pass or fail independently. Plus, [set_up](/test-files#set-up-function) and [tear_down](/test-files#tear-down-function) are called before and after each run. This reduces code repetition and helps create related tests more efficiently. +Under `--parallel` every row is dispatched as its own concurrent job, so rows sharing a +file, a temp path or an environment variable will race. Serialise them with the +`# bashunit: no-parallel-tests` directive at the top of the file (see +[Parallel](/command-line#parallel)), or give each row its own scratch path with +`bashunit::temp_file`. + +The annotation must be in the test file, but the provider function only has to be defined +by the time the run starts. Put shared providers in your bootstrap file (`--boot` or +`BASHUNIT_BOOTSTRAP`) and reference them by name from any test file: + +::: code-group +```bash [tests/bootstrap.sh] +function provider_supported_shells() { + bashunit::data_set "bash" + bashunit::data_set "zsh" +} +``` +```bash [tests/any_test.sh] +# @data_provider provider_supported_shells +function test_shell_is_available() { + assert_command_available "$1" +} +``` +::: + +::: warning A provider with no rows makes its test disappear +A provider that does not exist, or that emits no `bashunit::data_set` line, makes its test +run **zero** times, and bashunit does not error: the header still counts the test, nothing +runs, and the suite stays green. Compare the header count with the reported total (`Tests: 2` +in the header but `1 total` at the bottom means a provider produced no rows), or check the +selection with `./bashunit --list `. +::: + A data provider function is implemented as follows: ::: code-group @@ -151,6 +188,33 @@ Running example_test.sh ``` ::: +## Failing rows + +A failing row's result line is labelled with the test name only: the arguments are **not** +appended the way they are on a passing row. + +```[Output] +✓ Passed: Directory exists ('/usr') +✓ Passed: Directory exists ('/etc') +✗ Failed: Directory exists + Expected '/nope' + to exist but 'do not exist' + at example_test.sh:5 +``` + +Identify the row from the `Expected` value, or use `::1::` interpolation in the test title, +which does put the arguments in the name. + +## Combining with other options + +| Option | Effect on a provider | +|--------|----------------------| +| `--repeat ` | Every row runs n times; each row still reports one line | +| `--retry ` | Only the failing row is retried | +| `# @tag` / `--tag` | Tags belong to the test function, so they select all its rows | +| `--filter` | Matches the **function name**, not the interpolated title: `--filter directory_exists`, not `--filter "Directory exists"` | +| `--list` | Lists the function once; the row count is a property of the run (see [Command line](/command-line#list)) | + ## Related - [Test files](/test-files) — `set_up` and `tear_down` lifecycle hooks diff --git a/tests/functional/custom_asserts.sh b/tests/functional/custom_asserts.sh index 0fd4b11e..859fbc0a 100644 --- a/tests/functional/custom_asserts.sh +++ b/tests/functional/custom_asserts.sh @@ -4,7 +4,7 @@ function assert_foo() { local actual="$1" local expected="foo" - if [[ "$expected" != "$actual" ]]; then + if [ "$expected" != "$actual" ]; then bashunit::assertion_failed "$expected" "${actual}" return fi @@ -15,7 +15,7 @@ function assert_foo() { function assert_positive_number() { local actual="$1" - if [[ "$actual" -le 0 ]]; then + if [ "$actual" -le 0 ]; then bashunit::assertion_failed "positive number" "${actual}" "got" return fi From 35447798e79a421e60689867722d2a61e34c9dea Mon Sep 17 00:00:00 2001 From: Chemaclass Date: Tue, 11 Aug 2026 14:45:48 +0200 Subject: [PATCH 07/10] docs(installation): correct the action inputs, checksum modes and init effects The action's version input defaults to the release pinned at the ref, not latest, and its annotations input (added in #1042) was documented nowhere. The checksum tip gave the wrong reason for opting out and hid that the default is lenient: unset verifies and warns when verification is impossible, an explicit true aborts instead, false skips. The action always passes the variable, so its default is the strict mode. init also writes .github/workflows/tests.yml and creates or edits .env, commenting out an existing BASHUNIT_BOOTSTRAP line. Both quickstart and the command-line page said it only creates two files. Requirements now lists the optional tooling that silently changes behaviour, most importantly jq turning every JSON assertion into a skip. The pipeline tip pointed at tests.yml, which does not use the action at all; test-action.yml does. Adds an Updating section for bashunit upgrade, which shipped undocumented on this page. The nine-line WSL preamble was pasted four times; Requirements now carries it once. Quickstart's sample output showed a ./ the command did not pass and a duration column that only appears when the clock is cheap, and it never showed how to control a run. index.md's feature cards predated parallel runs, coverage, tags, sharding, --changed and every report format. README claimed 77 assertions; there are 84. --- README.md | 2 +- docs/index.md | 8 ++--- docs/installation.md | 83 ++++++++++++++++++++++++-------------------- docs/quickstart.md | 41 ++++++++++++++-------- 4 files changed, 77 insertions(+), 57 deletions(-) diff --git a/README.md b/README.md index f68f4e57..58b05657 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ ## Why bashunit A lightweight, fast testing framework for **Bash 3.0+**, focused on developer experience. -It ships 77 assertions plus spies, mocks, data providers, snapshots and more. +It ships 84 assertions plus spies, mocks, data providers, snapshots and more. ## Quick start diff --git a/docs/index.md b/docs/index.md index 802a4e03..808bb8e9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -25,14 +25,14 @@ features: - icon: src: /flexible.svg title: Flexible - details: Robust assertions for comparing, matching, and validating results, ensuring thorough testing of your codebase. + details: 84 assertions plus mocks, spies, data providers and snapshots, for comparing, matching and validating anything your scripts produce. - icon: src: /accessible.svg - title: Accessible - details: An intuitive API and clear documentation for a smooth developer experience, reducing testing complexity. + title: Fast and CI-ready + details: Run in parallel or shard the suite across runners, run only what changed, measure coverage, and publish JUnit, TAP, JSON, HTML or Markdown reports. - icon: src: /updated.svg - title: Updated + title: Community details: A vibrant GitHub community for support, collaboration, and continuous library enhancement. Join forces with like-minded developers. - icon: src: /multiplatform.svg diff --git a/docs/installation.md b/docs/installation.md index 9dc1ca00..607ae60c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -8,7 +8,18 @@ description: "Install bashunit via install.sh, npm, Brew, MacPorts or bashdep: a ## Requirements -bashunit requires **Bash 3.0** or newer. On Windows use [WSL](https://learn.microsoft.com/windows/wsl/install). +bashunit requires **Bash 3.0** or newer. + +On Windows, install [WSL](https://learn.microsoft.com/windows/wsl/install) (`wsl --install` +from an elevated PowerShell, then reboot) and run every command below inside the WSL shell. + +Everything else is optional, but some features need it: + +| Tool | Needed for | Without it | +|------|------------|------------| +| `jq` | [JSON assertions](/assertions#assert-json-key-exists) | the test is **skipped**, not failed | +| `git` | `--changed`, `--coverage-diff`, `upgrade`, failure diffs | those flags error out; diffs print plain values | +| `inotifywait` (Linux) / `fswatch` (macOS) | [`watch`](/command-line#watch) | falls back to polling | ## install.sh @@ -19,26 +30,21 @@ There is a tool that will generate an executable with the whole library in a sin curl -s https://bashunit.com/install.sh | bash ``` -```bash [Windows] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: -curl -s https://bashunit.com/install.sh | bash -``` ::: This will create a file inside a lib folder, such as `lib/bashunit`. ::: tip Automatic checksum verification -`install.sh` verifies the download against the release `checksum` asset by default and -aborts on a mismatch, so a tampered or corrupted download never lands. Set -`BASHUNIT_VERIFY_CHECKSUM=false` to opt out (e.g. for old releases published before -checksum assets existed). The manual check below is only needed when you opt out. +`install.sh` verifies the download against the release `checksum` asset and aborts on a +mismatch, so a tampered or corrupted download never lands. There are three states: + +- **unset** (default): verifies, aborts on a mismatch, and only *warns and continues* when + the checksum asset or a sha256 tool is unavailable +- `BASHUNIT_VERIFY_CHECKSUM=true`: also aborts when verification is impossible +- `BASHUNIT_VERIFY_CHECKSUM=false`: skips verification entirely + +The GitHub Action always passes the variable, so `verify-checksum: 'true'` (its default) is +the strict mode. The manual check below is only needed when you skip verification. ::: #### Verify @@ -64,17 +70,6 @@ The installation script can receive arguments (in any order): curl -s https://bashunit.com/install.sh | bash -s [dir] [version] ``` -```bash [Windows] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: -curl -s https://bashunit.com/install.sh | bash -s [dir] [version] -``` ::: - `[dir]`: the destiny directory to save the executable bashunit; `lib` by default @@ -236,14 +231,7 @@ bashdep::install "${DEPENDENCIES[@]}" ``` ```bash-vue [Windows - install-dependencies.sh] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: +# Run this inside the WSL shell, see Requirements above. # Ensure bashdep is installed [ ! -f lib/bashdep ] && { @@ -290,10 +278,11 @@ jobs: # For an immutable pin use a commit SHA: TypedDevs/bashunit@ # {{ pkg.version }} - uses: TypedDevs/bashunit@v0 with: - version: '{{ pkg.version }}' # or "latest" (default) + version: '{{ pkg.version }}' # omit for the version pinned at this ref directory: lib # optional, "lib" by default add-to-path: 'true' # optional, "true" by default verify-checksum: 'true' # optional, "true" by default + annotations: auto # optional, "auto" by default ("never" to turn off) # add-to-path puts the binary on $PATH, so just call "bashunit": - run: bashunit tests ``` @@ -343,7 +332,7 @@ jobs: ``` ::: -**Inputs:** `version` (default `latest`), `directory` (default `lib`), `add-to-path` (default `true`), `verify-checksum` (default `true`), `args` (default empty — when set, runs `bashunit ` after installing). +**Inputs:** `version` (default: the release pinned at this action ref, e.g. `0.45.0` on `@v0`; pass `latest` to always take the newest), `directory` (default `lib`), `add-to-path` (default `true`), `verify-checksum` (default `true`), `args` (default empty — when set, runs `bashunit ` after installing), `annotations` (default `auto` — GitHub Actions annotations for failing tests; `never` turns them off, `always` forces them). **Outputs:** `path` (binary path relative to the workspace), `version` (installed version). `verify-checksum` validates the downloaded binary against the release `checksum` @@ -387,9 +376,27 @@ Either way you get bashunit updates as routine pull requests — no manual re-pi `curl | bash` bumps to remember. Review the PR, let CI run, merge. ::: tip -See bashunit's own pipeline for a real example: https://github.com/TypedDevs/bashunit/blob/main/.github/workflows/tests.yml +See the action's own end-to-end test for a real example, covering `version`, `directory`, +`add-to-path` and `args`: +https://github.com/TypedDevs/bashunit/blob/main/.github/workflows/test-action.yml + +bashunit's `tests.yml` runs the in-repo entrypoint instead, so it is not an install example. ::: +## Updating + +For the `install.sh`, GitHub Action and bashdep routes, upgrade the binary in place: + +```bash +./lib/bashunit upgrade +``` + +It downloads the newest release over the same file, and prints +`> You are already on latest version` when there is nothing to do. + +Package-manager installs update through their own manager instead: `brew upgrade bashunit`, +`sudo port upgrade bashunit`, `npm install --save-dev bashunit@latest`. + ## Shell completion bashunit ships tab-completion scripts for bash and zsh under diff --git a/docs/quickstart.md b/docs/quickstart.md index f1806e6e..8bc11ce0 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -30,17 +30,6 @@ npm install -g bashunit bashunit tests/ ``` -```bash [Windows] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: -curl -s https://bashunit.com/install.sh | bash -``` ::: The `install.sh` route creates `lib/bashunit`; the npm route exposes `bashunit` via `npx` or your global `PATH`. @@ -55,7 +44,15 @@ You can bootstrap a ready to use test suite with the `init` subcommand: ./lib/bashunit init tests ``` -This will create a `tests` directory containing a sample test and bootstrap file. +It creates, in the current directory: + +- `tests/bootstrap.sh` — sourced before your tests; put shared setup here +- `tests/example_test.sh` — a sample test +- `.github/workflows/tests.yml` — a CI workflow using the official action +- `.env` — with `BASHUNIT_BOOTSTRAP=tests/bootstrap.sh`, which is what makes the bootstrap load + +If `.env` already sets `BASHUNIT_BOOTSTRAP`, `init` comments that line out and appends the +new one, so check the diff before committing. Alternatively, create your tests manually: @@ -77,14 +74,14 @@ Alternatively, create your tests manually: 3. Finally, run the **bashunit** executable: ```bash - ./lib/bashunit ./tests + ./lib/bashunit tests ``` 4. If everything works correctly, you should see an output similar to the following: ```-vue bashunit - {{ pkg.version }} | Tests: 1 Running tests/example_test.sh - ✓ Passed: Bashunit is working 16ms + ✓ Passed: Bashunit is working Tests: 1 passed, 1 total Assertions: 1 passed, 1 total @@ -93,8 +90,22 @@ Alternatively, create your tests manually: Time taken: 90ms ``` + A per-test duration column appears when the platform has a cheap clock source; + `BASHUNIT_SHOW_EXECUTION_TIME` controls it. + 5. Now you can start testing the functionalities of your own Bash scripts. +## Running your suite + +With no path, bashunit runs `tests/` (`BASHUNIT_DEFAULT_PATH`): + +```bash +./lib/bashunit # run tests/ +./lib/bashunit --filter user # only tests whose name matches +./lib/bashunit --parallel tests/ # run files concurrently +./lib/bashunit --changed # only test files touched since origin/HEAD +``` + ## Learning bashunit interactively If you prefer hands-on learning, bashunit includes an interactive tutorial: @@ -114,6 +125,8 @@ Dive deeper into the documentation: - **[Data providers](/data-providers)** - Write parameterized tests efficiently - **[Snapshots](snapshots)** - Test complex output easily - **[Test files](/test-files)** - Understand test file structure and lifecycle hooks +- **[Command line](/command-line)** - Every flag, from `--filter` to `--shard` +- **[Configuration](/configuration)** - `.env` and `BASHUNIT_*` variables