diff --git a/.env.example b/.env.example index 0eb11ea5..9fbee545 100644 --- a/.env.example +++ b/.env.example @@ -35,6 +35,9 @@ BASHUNIT_PROFILE= # Default: false (report slowest tests after BASHUNIT_PROFILE_COUNT= # Default: 10 (how many slowest tests to report) BASHUNIT_NO_COLOR= # Default: false (disable colors) BASHUNIT_NO_DIFF= # Default: false (disable unified diff on multiline assert failures) +BASHUNIT_NO_PROGRESS= # Default: false (suppress real-time progress, final results only) +BASHUNIT_SHOW_OUTPUT_ON_FAILURE= # Default: true (show captured test output on failure) +BASHUNIT_OUTPUT_FORMAT= # Default: empty (tap = TAP version 13 on stdout) #─────────────────────────────────────────────────────────────────────────────── # Test Execution @@ -49,13 +52,22 @@ BASHUNIT_GHA_ANNOTATIONS= # Default: auto (or always, never) BASHUNIT_REPORT_MD= # Default: empty (Markdown summary path) BASHUNIT_CHANGED= # Default: false (run only test files changed since a git ref) BASHUNIT_CHANGED_REF= # Default: empty (--changed ref: origin/HEAD, then HEAD) -BASHUNIT_COVERAGE_DIFF= # Default: empty (restrict coverage to lines changed since this ref) +BASHUNIT_COVERAGE_DIFF= # Default: empty (restrict coverage to lines changed since this ref) BASHUNIT_EXCLUDE_FILTER= # Default: empty (skip tests whose name matches) BASHUNIT_LIST_TESTS= # Default: false (print the tests that would run, run none) BASHUNIT_LIST_FORMAT= # Default: text (--list rendering: text or json) BASHUNIT_STOP_ON_ASSERTION_FAILURE= # Default: true (stop test on first assertion fail) BASHUNIT_STRICT_MODE= # Default: false (enable set -euo pipefail) BASHUNIT_LOGIN_SHELL= # Default: false (source login shell profiles) +BASHUNIT_PARALLEL_JOBS= # Default: 0 (unlimited; N or auto caps concurrency) +BASHUNIT_RANDOM_ORDER= # Default: false (shuffle files and tests) +BASHUNIT_SEED= # Default: empty (pins --random-order for a replay) +BASHUNIT_RETRY= # Default: 0 (re-run a failed test up to N extra times) +BASHUNIT_TEST_TIMEOUT= # Default: 0 (off; fail a test running longer than N seconds) +BASHUNIT_SHARD_INDEX= # Default: empty (run shard i of BASHUNIT_SHARD_TOTAL) +BASHUNIT_SHARD_TOTAL= # Default: empty (how many shards to split the suite into) +BASHUNIT_SKIP_ENV_FILE= # Default: false (skip .env and .bashunitrc) +BASHUNIT_WATCH_INTERVAL= # Default: 3 (seconds between polls in the watch fallback) #─────────────────────────────────────────────────────────────────────────────── # Reports @@ -63,6 +75,15 @@ BASHUNIT_LOGIN_SHELL= # Default: false (source login shell profile BASHUNIT_LOG_JUNIT= # JUnit XML report path (e.g., report.xml) BASHUNIT_LOG_GHA= # GitHub Actions workflow-commands log path (e.g., gha.log) BASHUNIT_REPORT_HTML= # HTML test report path (e.g., report.html) +BASHUNIT_REPORT_TAP= # TAP version 13 report path (e.g., report.tap) +BASHUNIT_REPORT_JSON= # JSON report path (e.g., report.json) + +#─────────────────────────────────────────────────────────────────────────────── +# Snapshots +#─────────────────────────────────────────────────────────────────────────────── +BASHUNIT_SNAPSHOT_UPDATE= # Default: false (rewrite existing snapshots) +BASHUNIT_SNAPSHOT_CREATE= # Default: true (record a missing snapshot instead of failing) +BASHUNIT_SNAPSHOT_REPORT_UNUSED= # Default: false (list snapshots no test resolved) #─────────────────────────────────────────────────────────────────────────────── # Code Coverage @@ -74,7 +95,11 @@ BASHUNIT_COVERAGE_REPORT= # Default: coverage/lcov.info BASHUNIT_COVERAGE_REPORT_HTML= # HTML coverage report directory (e.g., coverage/html) BASHUNIT_COVERAGE_MIN= # Minimum coverage % (fails if below) BASHUNIT_COVERAGE_THRESHOLD_LOW= # Default: 50 (red below this) -BASHUNIT_COVERAGE_THRESHOLD_HIGH= # Default: 80 (green above this) +BASHUNIT_COVERAGE_THRESHOLD_HIGH= # Default: 80 (green at or above this) +BASHUNIT_COVERAGE_ENGINE= # Default: auto (or xtrace, trap) +BASHUNIT_COVERAGE_SHOW_LINE_HITS= # Default: false (print per-line execution counts) +BASHUNIT_COVERAGE_SHOW_FUNCTIONS= # Default: false (print per-function coverage) +BASHUNIT_COVERAGE_SHOW_UNCOVERED= # Default: false (print missed line ranges per file) #─────────────────────────────────────────────────────────────────────────────── # Advanced / Debug @@ -82,3 +107,4 @@ BASHUNIT_COVERAGE_THRESHOLD_HIGH= # Default: 80 (green above this) BASHUNIT_DEV_LOG= # Developer log file path BASHUNIT_BENCH_MODE= # Default: false (benchmark mode) BASHUNIT_INTERNAL_LOG= # Default: false (internal debug logging) +BASHUNIT_NO_DEPRECATION_WARNINGS= # Default: false (silence unprefixed-alias warnings) diff --git a/CHANGELOG.md b/CHANGELOG.md index b1c1cbaf..787e4de5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,28 +3,33 @@ ## Unreleased ### Added -- `--coverage-diff ` restricts the coverage console report to lines changed since a base ref, and `--coverage-min` then gates on that diff percentage (#1032) -- `--report-md ` writes a Markdown run summary: verdict, counts table, failures with their message, plus coverage and slowest tests when those ran. Inside GitHub Actions it is appended to `$GITHUB_STEP_SUMMARY` automatically, so failures render on the job page (#1015) -- `--gha-annotations ` controls GitHub Actions annotations on stdout; `auto` turns them on inside GitHub Actions and stays quiet everywhere else (#1014) -- `--repeat ` runs each selected test n times so flakiness can be hunted before it reaches CI. The test is reported once with the aggregate outcome, a failure names the iteration it happened on, and repeat wraps `--retry` rather than the other way round (#1013) -- Flaky is a first-class outcome: a test that only passed after a retry is counted separately, kept inside the pass total so the exit code is unchanged, and carried into JUnit (``), TAP, JSON, HTML and GitHub Actions along with the first attempt's failure message. `--fail-on-flaky` turns such a run red (#1012) -- `--order-by ` picks the execution order: `defined` (default), `defects` (last run's failures first, whole suite still runs) or `random`. `--random-order` and `--seed` keep working unchanged (#1011) -- `--changed []` runs only the test files git reports as touched since `` (default `origin/HEAD`, then `HEAD`), covering committed, staged, unstaged and untracked changes. Deletions are dropped, a rename selects its new path, and a missing work tree or unresolvable ref fails the run instead of selecting nothing (#1010) +- `--coverage-diff ` limits the coverage console report to lines changed since a base ref; `--coverage-min` then gates on that diff percentage (#1032) +- `assert_command_available ` asserts a command, shell builtin or function resolves through `command -v` (#1027) - `assert_between ` and `assert_not_between` add inclusive numeric-range assertions for integers and decimals (#1026) -- `--list` (alias `--dry-run`) prints the tests a run would execute, without running them; `--list-format json` emits file, function, name, line and tags. Honours every selection flag, including `--shard` and `--random-order --seed` ordering (#1007) +- `--report-md ` writes a Markdown run summary: verdict, counts table, failures with their message, plus coverage and slowest tests when those ran. Inside GitHub Actions it is also appended to `$GITHUB_STEP_SUMMARY` (#1015) +- `--gha-annotations ` controls GitHub Actions annotations on stdout; `auto` enables them only inside GitHub Actions (#1014) +- `--repeat ` runs each selected test n times to hunt flakiness before CI does. One report line with the aggregate outcome, a failure names its iteration, and repeat wraps `--retry` (#1013) +- Flaky is a first-class outcome: a test that only passed after a retry is counted separately, stays inside the pass total so the exit code is unchanged, and is carried into JUnit (``), TAP, JSON, HTML and GitHub Actions with the first attempt's failure message. `--fail-on-flaky` turns such a run red (#1012) +- `--order-by ` picks the execution order: `defined` (default), `defects` (last run's failures first, whole suite still runs) or `random`. `--random-order` and `--seed` are unchanged (#1011) +- `--changed []` runs only the test files git reports as touched since `` (default `origin/HEAD`, then `HEAD`), covering committed, staged, unstaged and untracked changes. Deletions are dropped, a rename selects its new path, and a missing work tree or unresolvable ref fails the run (#1010) - `--exclude-filter ` skips tests by name, the counterpart of `--exclude-tag`. Repeatable, OR'd, and wins over `--filter` (#1009) -- `# @tags a b` above any top-level line applies those tags to every test in the file, unioned with per-function `# @tag` (#1008) - `--tag` accepts expressions: `'a&&b'` (AND) and `'!a'` (NOT), combinable as `'a&&!b'`. Repeated `--tag` flags keep OR semantics, and `--exclude-tag` still wins (#1008) -- `assert_command_available ` asserts that an external command, shell builtin or function resolves through `command -v` (#1027) -- The coverage engine in use is reported by `--verbose`, and an explicit `BASHUNIT_COVERAGE_ENGINE=xtrace` that the running Bash cannot honour now warns instead of being silently ignored (#1005) +- `# @tags a b` above any top-level line tags every test in the file, unioned with per-function `# @tag` (#1008) +- `--list` (alias `--dry-run`) prints the tests a run would execute without running them; `--list-format json` emits file, function, name, line and tags. Honours every selection flag, including `--shard` and `--random-order --seed` ordering (#1007) +- `--verbose` reports the coverage engine in use, and an explicit `BASHUNIT_COVERAGE_ENGINE=xtrace` the running Bash cannot honour now warns instead of being silently ignored (#1005) ### Changed -- The JUnit XML shape changed: one `` per test file (with its own counts, time and timestamp) instead of a single flat suite, `classname` on every ``, `` carrying the first informative line of the real message with `type="AssertionFailed"`, `` with the test's captured output, and aggregate totals on ``. Consumers that group by suite or classname (Jenkins, GitLab, dorny/test-reporter) now get real groupings (#1016) -- Performance: `--coverage` is about 1.6x to 2.3x faster. Executable-line classification no longer forks `grep` per source line, which was roughly half of a coverage run's wall time and affected both engines equally (#1005) +- `bashunit test --help` lists `--show-skipped` and `--show-incomplete`, both accepted by the parser but never advertised. `BASHUNIT_COVERAGE_SHOW_FUNCTIONS` and `BASHUNIT_COVERAGE_SHOW_UNCOVERED` are registered in `src/config/env.sh` like every other setting, and `.env.example` now lists all 66 settings, 19 of which were missing +- Docs: a full audit of the reference pages against the code. `docs/configuration.md` gained the 17 settings it never documented, `docs/command-line.md` gained the `assert` subcommand section and had every stale example output replaced with a real one, the `assert_equals` examples in `docs/assertions.md` were inverted, and the coverage settings and diff-coverage narrative now live in one place instead of two that had drifted +- JUnit XML: one `` per test file with its own counts, time and timestamp instead of a single flat suite, `classname` on every ``, `` carrying the first informative line of the real message with `type="AssertionFailed"`, `` with the test's captured output, and aggregate totals on ``. Consumers that group by suite or classname (Jenkins, GitLab, dorny/test-reporter) now get real groupings (#1016) +- Performance: `--coverage` is about 1.6x to 2.3x faster. Executable-line classification no longer forks `grep` per source line, roughly half of a coverage run's wall time on both engines (#1005) ### Fixed +- A failed assertion is no longer reported twice. `bashunit::assert_that` returns 1 on failure by design, so a custom assertion ending with it made the test body exit non-zero and the runner printed a spurious `✗ Error` on top of the `✗ Failed`. Custom assertions no longer need a trailing `return 0`; a real runtime error is still reported as an error +- `--coverage-report` with no value uses `coverage/lcov.info` instead of aborting the run with `$2: unbound variable`, and no longer consumes a following flag as its filename. Write the test path before it, since an optional value cannot be told apart from a path +- `--list --list-format json` reports each tag as its own array element. The emitter split the tag list on whitespace while every other consumer splits it on commas, so a test with two tags rendered as `["slow,fileTag"]` and a tag containing spaces was split into one element per word - Build: the standalone binary size budget is 544 KiB, raised from 500 KiB after ordinary feature growth crossed it; the artifact keeps its indentation rather than being minified (#1045) -- `assert_within_delta` rejects malformed numbers such as `1.2.3` or `5-3` as non-numeric instead of leaking a raw `bc` parse error or silently evaluating them as an expression (#1026) +- `assert_within_delta` rejects malformed numbers such as `1.2.3` or `5-3` as non-numeric instead of leaking a raw `bc` parse error or evaluating them as an expression (#1026) - Report formats are no longer empty under `--parallel`. `--report-junit`, `--report-tap`, `--report-json`, `--report-html` and `--log-junit` all recorded zero tests, because the rows were collected inside the per-test worker and nothing rebuilt them in the parent (#1004) ## [0.45.0](https://github.com/TypedDevs/bashunit/compare/0.44.0...0.45.0) - 2026-08-09 diff --git a/README.md b/README.md index f68f4e57..58b05657 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ ## Why bashunit A lightweight, fast testing framework for **Bash 3.0+**, focused on developer experience. -It ships 77 assertions plus spies, mocks, data providers, snapshots and more. +It ships 84 assertions plus spies, mocks, data providers, snapshots and more. ## Quick start diff --git a/docs/assertions.md b/docs/assertions.md index 29ce8a79..bbc53c51 100644 --- a/docs/assertions.md +++ b/docs/assertions.md @@ -14,6 +14,18 @@ other helper does take the `bashunit::` prefix; see [Globals](/globals). Run `bashunit doc` to print this catalogue in your terminal, or `bashunit doc ` to narrow it (`bashunit doc json`). +Any assertion here also runs from the shell without a test file: +`bashunit assert contains "world" "hello world"`. See [Standalone](/standalone). + +`assert_same`, `assert_equals`, `assert_not_same`, `assert_not_equals` and the numeric +comparisons take an optional last argument used as the failure label: +`assert_same "1" "2" "my custom label"` reports `✗ Failed: my custom label`. + +The string assertions whose signature ends in `...` take a variadic value: every argument +after the first is joined with **newlines** and compared as one value. They accept no +trailing label override, so `assert_contains "zzz" "abc" "my label"` searches +`abc\nmy label` instead of relabelling the failure. + ## Quick reference | Group | Assertions | @@ -55,9 +67,10 @@ above. A purpose-built assertion is usually clearer still — `assert_directory_exists` rather than a hand-rolled `test -d`. -Reports an error if the argument result in a truthy value: `true` or `0`. +Reports an error **unless** the argument results in a truthy value: `true` or `0`, or a +command or function that exits `0`. -- [assert_false](#assert-false) is similar but different. +- [assert_false](#assert-false) is the inverse of this assertion and takes the same arguments. ::: code-group ```bash [Example] @@ -88,9 +101,10 @@ function mock_false() { ## assert_false > `assert_false bool|function|command [args...]` -Reports an error if the argument result in a falsy value: `false` or `1`. +Reports an error **unless** the argument results in a falsy value: `false` or `1`, or a +command or function that exits non-zero. -- [assert_true](#assert-true) is similar but different. +- [assert_true](#assert-true) is the inverse of this assertion and takes the same arguments. ::: code-group ```bash [Example] @@ -148,17 +162,17 @@ Reports an error if the two variables `expected` and `actual` are not equal igno ::: code-group ```bash [Example] function test_success() { - assert_equals "foo" "\e[31mfoo" + assert_equals "foo" $'\e[31mfoo' } function test_failure() { - assert_equals "\e[31mfoo" "\e[31mfoo" + assert_equals "foo" $'\e[31mbar' } ``` ::: ## assert_contains -> `assert_contains "needle" "haystack"` +> `assert_contains "needle" "haystack"...` Reports an error if `needle` is not a substring of `haystack`. @@ -214,10 +228,14 @@ function test_failure() { ::: ## assert_matches -> `assert_matches "pattern" "value"` +> `assert_matches "pattern" "value"...` Reports an error if `value` does not match the regular expression `pattern`. +`pattern` is an ERE evaluated by `grep -E`. If it does not match as written, the value is +retried with every newline replaced by a space, so a single-line pattern can match across +lines: `assert_matches 'one two'` matches the two-line value `one\ntwo`. + - [assert_not_matches](#assert-not-matches) is the inverse of this assertion and takes the same arguments. ::: code-group @@ -233,7 +251,7 @@ function test_failure() { ::: ## assert_string_starts_with -> `assert_string_starts_with "needle" "haystack"` +> `assert_string_starts_with "needle" "haystack"...` Reports an error if `haystack` does not starts with `needle`. @@ -252,7 +270,7 @@ function test_failure() { ::: ## assert_string_ends_with -> `assert_string_ends_with "needle" "haystack"` +> `assert_string_ends_with "needle" "haystack"...` Reports an error if `haystack` does not ends with `needle`. @@ -280,7 +298,7 @@ Reports an error if `value` does not match the `format` string. The format strin | `%d` | One or more digits | | `%i` | Signed integer (e.g. `+1`, `-42`) | | `%f` | Floating point number (e.g. `3.14`) | -| `%s` | One or more non-whitespace characters | +| `%s` | One or more characters other than a space (tabs and newlines match) | | `%x` | Hexadecimal (e.g. `ff00ab`) | | `%e` | Scientific notation (e.g. `1.5e10`) | | `%%` | Literal `%` character | @@ -301,10 +319,13 @@ function test_failure() { ::: ## assert_line_count -> `assert_line_count "count" "haystack"` +> `assert_line_count "count" "haystack"...` Reports an error if `haystack` does not contain `count` lines. +A literal `\n` (backslash followed by `n`) counts as a line break too, so +`assert_line_count 2 'one\ntwo'` passes even though the value holds no real newline. + ::: code-group ```bash [Example] function test_success() { @@ -345,7 +366,7 @@ function test_failure() { Reports an error if `actual` is not less than or equal to `expected`. -- [assert_greater_than](#assert-greater-or-equal-than) is the inverse of this assertion and takes the same arguments. +- [assert_greater_or_equal_than](#assert-greater-or-equal-than) is the counterpart of this assertion and takes the same arguments. ::: code-group ```bash [Example] @@ -411,6 +432,10 @@ function test_failure() { Reports an error if `actual` is outside the inclusive numeric range from `min` to `max`. Integers, decimals, and negative values are supported. `min` must not be greater than `max`. +A non-numeric argument, or `min` greater than `max`, is a **usage error**: the assertion +returns 2 and the test is reported as an Error rather than a failure, with a message such +as `assert_between expects min <= max, got '500' and '100'`. + - [assert_not_between](#assert-not-between) is the exact negation and takes the same arguments. ::: code-group @@ -432,6 +457,10 @@ function test_failure() { Reports an error if `actual` is inside the inclusive numeric range from `min` to `max`. Integers, decimals, and negative values are supported. `min` must not be greater than `max`. +A non-numeric argument, or `min` greater than `max`, is a **usage error**: the assertion +returns 2 and the test is reported as an Error rather than a failure, with a message such +as `assert_between expects min <= max, got '500' and '100'`. + - [assert_between](#assert-between) is the exact negation and takes the same arguments. ::: code-group @@ -453,6 +482,10 @@ Reports an error if `actual` is not within `delta` of `expected` (i.e. `|actual - expected| > delta`). Supports floating-point values. Useful for timing or measured values where exact equality is too strict. +The bound is inclusive, so `|actual - expected| == delta` passes. An operand that is not a +number, such as `1.2.3` or `5-3`, fails the assertion with `to all be numeric` instead of +being evaluated as an expression. A leading `+` is accepted. + ::: code-group ```bash [Example] function test_success() { @@ -480,6 +513,10 @@ Inputs are automatically converted to epoch seconds. Supported formats: You can mix formats in the same assertion (e.g., one epoch, one ISO). +Anything else, including an empty string, fails the assertion with +`Expected '' to be 'a valid date'` rather than being coerced to `0`. This applies to +all five date assertions. + ::: code-group ```bash [Example] function test_success() { @@ -500,7 +537,7 @@ function test_failure() { Reports an error if `actual` is not before `expected` (i.e. `actual` must be less than `expected`). -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -519,7 +556,7 @@ function test_failure() { Reports an error if `actual` is not after `expected` (i.e. `actual` must be greater than `expected`). -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -538,7 +575,7 @@ function test_failure() { Reports an error if `actual` does not fall between `from` and `to` (inclusive). -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -557,7 +594,10 @@ function test_failure() { Reports an error if `actual` is not within `delta` seconds of `expected`. -Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert_date_equals) for supported formats. +`delta` is required: omitting it produces a shell error reported as an Error, not an +assertion failure. + +Inputs are automatically converted to epoch seconds. See [assert_date_equals](#assert-date-equals) for supported formats. ::: code-group ```bash [Example] @@ -634,6 +674,11 @@ Use `--stdin` to feed input into interactive commands (e.g. commands using Use `--stdout-contains` / `--stdout-not-contains` (and the `stderr-*` variants) for substring matching when you don't want to assert against the full output. +Unrecognised arguments are silently dropped, so a mistyped flag such as +`--stdout-contain` checks nothing and the assertion passes on exit status alone. Extra +words after the command are dropped too: put arguments inside the command string, +`assert_exec "echo hello" --stdout "hello"`. + ::: code-group ```bash [Example] function sample() { @@ -697,7 +742,12 @@ function test_failure() { ## assert_array_contains > `assert_array_contains "needle" "haystack"` -Reports an error if `needle` is not an element of `haystack`. +Reports an error if `needle` is not found in `haystack`. + +`needle` is matched as a **substring of the array joined with spaces**, not element by +element, so `assert_array_contains "oob" foobar baz` and +`assert_array_contains "foo bar" foo bar baz` both pass. Use +[assert_arrays_equal](#assert-arrays-equal) when you need exact element comparison. - [assert_array_not_contains](#assert-array-not-contains) is the inverse of this assertion and takes the same arguments. @@ -960,7 +1010,9 @@ function test_failure() { ## assert_file_contains > `assert_file_contains "file" "search"` -Reports an error if `file` does not contains the search string. +Reports an error if `file` does not contain the search string. + +`search` is matched **literally** (`grep -F`); regex metacharacters have no special meaning. - [assert_file_not_contains](#assert-file-not-contains) is the inverse of this assertion and takes the same arguments. @@ -1158,8 +1210,8 @@ Reports an error if `directory` is not an empty directory. ::: code-group ```bash [Example] function test_success() { - local directory="/home/user/empty_directory" - mkdir "$directory" + local directory + directory="$(bashunit::temp_dir)" assert_is_directory_empty "$directory" } @@ -1188,7 +1240,8 @@ function test_success() { } function test_failure() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -r "$directory" assert_is_directory_readable "$directory" @@ -1212,7 +1265,8 @@ function test_success() { } function test_failure() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -w "$directory" assert_is_directory_writable "$directory" @@ -1299,7 +1353,7 @@ function test_failure() { ::: ## assert_not_contains -> `assert_not_contains "needle" "haystack"` +> `assert_not_contains "needle" "haystack"...` Reports an error if `needle` is a substring of `haystack`. @@ -1318,7 +1372,7 @@ function test_failure() { ::: ## assert_string_not_starts_with -> `assert_string_not_starts_with "needle" "haystack"` +> `assert_string_not_starts_with "needle" "haystack"...` Reports an error if `haystack` does starts with `needle`. @@ -1337,7 +1391,7 @@ function test_failure() { ::: ## assert_string_not_ends_with -> `assert_string_not_ends_with "needle" "haystack"` +> `assert_string_not_ends_with "needle" "haystack"...` Reports an error if `haystack` does ends with `needle`. @@ -1375,7 +1429,7 @@ function test_failure() { ::: ## assert_not_matches -> `assert_not_matches "pattern" "value"` +> `assert_not_matches "pattern" "value"...` Reports an error if `value` matches the regular expression `pattern`. @@ -1415,7 +1469,11 @@ function test_failure() { ## assert_array_not_contains > `assert_array_not_contains "needle" "haystack"` -Reports an error if `needle` is an element of `haystack`. +Reports an error if `needle` is found in `haystack`. + +`needle` is matched as a **substring of the array joined with spaces**, not element by +element, so `assert_array_not_contains "foo bar" foo bar` fails even though no single +element is `foo bar`. - [assert_array_contains](#assert-array-contains) is the inverse of this assertion and takes the same arguments. @@ -1467,6 +1525,10 @@ function test_failed() { Reports an error if `file` contains the search string. +`search` is matched as a **basic regular expression** (`grep`), unlike +[assert_file_contains](#assert-file-contains) which matches literally, so +`assert_file_not_contains file 'a.c'` fails on a file containing `abc`. + - [assert_file_contains](#assert-file-contains) is the inverse of this assertion and takes the same arguments. ::: code-group @@ -1526,8 +1588,8 @@ function test_success() { } function test_failure() { - local directory="/home/user/empty_directory" - mkdir "$directory" + local directory + directory="$(bashunit::temp_dir)" assert_is_directory_not_empty "$directory" } @@ -1544,7 +1606,8 @@ Reports an error if `directory` is readable. ::: code-group ```bash [Example] function test_success() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -r "$directory" assert_is_directory_not_readable "$directory" @@ -1568,7 +1631,8 @@ Reports an error if `directory` is writable. ::: code-group ```bash [Example] function test_success() { - local directory="/home/user/test" + local directory + directory="$(bashunit::temp_dir)" chmod -w "$directory" assert_is_directory_not_writable "$directory" @@ -1586,7 +1650,7 @@ function test_failure() { ## assert_files_not_equals > `assert_files_not_equals "expected" "actual"` -Reports an error if `expected` and `actual` are not equals. +Reports an error if `expected` and `actual` have the same contents. - [assert_files_equals](#assert-files-equals) is the inverse of this assertion and takes the same arguments. @@ -1627,6 +1691,10 @@ function test_failure() { Reports an error if `key` does not exist in the JSON string. Uses [jq](https://jqlang.github.io/jq/) syntax for key paths. Requires `jq` to be installed; if missing the test is skipped. +A key whose value is `null` or `false` is reported as missing, because `jq -e` treats both +as absent. `0` and `""` are fine. To assert a false or null value, use +`assert_json_contains ".flag" "false" "$json"`. + ::: code-group ```bash [Example] function test_success() { @@ -1645,6 +1713,9 @@ function test_failure() { Reports an error if `key` does not exist in the JSON string or its value does not equal `expected`. Uses [jq](https://jqlang.github.io/jq/) syntax for key paths. Requires `jq` to be installed; if missing the test is skipped. +A key whose value is `null` or `false` is reported as missing, the same guard +[assert_json_key_exists](#assert-json-key-exists) uses. + ::: code-group ```bash [Example] function test_success() { @@ -1681,6 +1752,10 @@ function test_failure() { Reports an error if `command` takes longer than `threshold_ms` milliseconds to execute. Uses the framework's portable clock internally. +Requires `awk`, plus `bc` or `awk` for the arithmetic. Unlike the JSON assertions, the +duration assertions do **not** skip when the tool is missing: the test is reported as an +Error. + ::: code-group ```bash [Example] function test_success() { @@ -1798,6 +1873,11 @@ function test_colored_render_modes() { Reports an error if the spied `command` was never called. Requires `bashunit::spy command` first — see [Test doubles](/test-doubles). +Every `assert_have_been_called*` and `assert_not_called` fails with +`was never registered as a spy` when the name was never passed to `bashunit::spy`, +including `assert_not_called`, which does **not** pass for an unspied name. Spies are +cleared between tests, so spy inside the test that asserts on it. + ::: code-group ```bash [Example] function test_success() { @@ -1818,7 +1898,12 @@ function test_failure() { ## assert_have_been_called_with > `assert_have_been_called_with "command" "expected_args" [nth]` -Reports an error if the spied `command` was not called with `expected_args`. Checks the **last** call unless a trailing all-digits `nth` selects a specific one; the failure names the call it compared. To match any call, use [assert_have_been_called_with_any](#assert-have-been-called-with-any). +Reports an error if the spied `command` was not called with `expected_args`. Checks the **last** call unless a trailing all-digits `nth` selects a specific one; the failure names the call it compared. +Because `nth` is detected as a trailing all-digits argument, an expectation whose last +argument is a number is read as the selector: `assert_have_been_called_with git commit -m 5` +compares `commit -m` against call 5. Pass the expectation as one quoted string, +`assert_have_been_called_with git "commit -m 5"`, or use +[assert_have_been_called_with_args](#assert-have-been-called-with-args), which has no `nth`. To match any call, use [assert_have_been_called_with_any](#assert-have-been-called-with-any). Note the argument order: the spy comes first here, but *second* in [assert_have_been_called_times](#assert-have-been-called-times). @@ -2045,3 +2130,4 @@ function test_failure() { - [Test doubles](/test-doubles) — mocks and spies for isolated tests - [Data providers](/data-providers) — run the same assertions over many inputs - [Globals](/globals) — `bashunit::` helper functions +- [Standalone](/standalone) — run these assertions straight from the command line diff --git a/docs/command-line.md b/docs/command-line.md index 972e7f69..bbaf9ccb 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -58,7 +58,7 @@ bashunit test tests/ --parallel --simple | Option | Description | |--------------------------------|--------------------------------------------------| -| `-a, --assert ` | Run a standalone assert function | +| `-a, --assert ` | Run a standalone assert function (deprecated: use [`bashunit assert`](#assert)) | | `-e, --env, --boot ` | Load custom env/bootstrap file (supports args) | | `-f, --filter ` | Only run tests matching name | | `--exclude-filter ` | Skip tests whose name matches (repeatable) | @@ -110,19 +110,18 @@ bashunit test tests/ --parallel --simple | `--skip-env-file` | Skip `.env` loading, use shell environment only | | `-l, --login` | Run tests in login shell context | | `--no-color` | Disable colored output | -| `--coverage` | Enable code coverage tracking | -| `--coverage-paths ` | Paths to track (default: auto-discover) | -| `--coverage-exclude ` | Exclusion patterns | -| `--coverage-report [file]` | LCOV output path (default: `coverage/lcov.info`) | -| `--coverage-report-html [dir]` | Generate HTML report (default: `coverage/html`) | -| `--coverage-min ` | Minimum coverage threshold | -| `--coverage-diff ` | Report coverage only for lines changed since ref | -| `--no-coverage-report` | Console output only, no LCOV file | +| `-h, --help` | Show the test help | +| `--coverage*` | Eight coverage flags, see [Coverage](#coverage) | ### Standalone Assert > `bashunit test -a|--assert function "arg1" "arg2"` +::: warning Deprecated +Use the [`assert` subcommand](#assert) instead. This form still works and prints a +deprecation notice on stderr. +::: + Run a core assert function standalone without a test context. ::: code-group @@ -130,7 +129,8 @@ Run a core assert function standalone without a test context. bashunit test --assert equals "foo" "bar" ``` ```[Output] -✗ Failed: Main::exec assert +Deprecated: `bashunit test --assert`. Use `bashunit assert` instead. +✗ Failed: assert equals Expected 'foo' but got 'bar' ``` @@ -485,8 +485,10 @@ developer actually looks at first: `tests/math_test.sh:42` ``` -Expected '4' -but got '5' +✗ Failed: Sums two numbers + Expected '4' + but got '5' + at tests/math_test.sh:42 ``` ``` @@ -515,7 +517,7 @@ Inside GitHub Actions, bashunit annotates failing tests on the pull request by itself. No flag, no configuration: ``` -::error file=tests/math_test.sh,line=42,title=Sums::Expected '4' but got '5' +::error file=tests/math_test.sh,line=42,title=Sums two numbers::✗ Failed: Sums two numbers%0A Expected '4'%0A but got '5'%0A at tests/math_test.sh:42 ``` GitHub parses workflow commands from the **job log**, so the annotations go to @@ -548,15 +550,15 @@ The `--report-json` flag writes machine-readable results for scripts, dashboards ```json { - "summary": { "total": 3, "passed": 2, "failed": 1, "skipped": 0, "incomplete": 0, "duration_ms": 42 }, + "summary": { "total": 3, "passed": 2, "failed": 1, "skipped": 0, "incomplete": 0, "flaky": 0, "duration_ms": 42 }, "tests": [ - { "file": "tests/math_test.sh", "name": "it adds", "status": "passed", "duration_ms": 5, "message": "" }, - { "file": "tests/math_test.sh", "name": "it divides", "status": "failed", "duration_ms": 3, "message": "Expected 2 but got 3" } + { "file": "tests/math_test.sh", "name": "it adds", "status": "passed", "duration_ms": 5, "retries": 0, "message": "" }, + { "file": "tests/math_test.sh", "name": "it divides", "status": "failed", "duration_ms": 3, "retries": 0, "message": "Expected 2 but got 3" } ] } ``` -`status` is one of `passed`, `failed`, `skipped`, `incomplete` (`snapshot` and `risky` are also emitted per test and counted as passed in the summary). Like the other file reporters, per-test rows come from a sequential run; under `--parallel` the file is still valid JSON. +`status` is one of `passed`, `failed`, `skipped`, `incomplete`, `flaky` (`snapshot` and `risky` are also emitted per test and counted as passed in the summary). Per-test rows are complete in both modes; under `--parallel` the row order follows completion order rather than definition order. ### Show Output on Failure @@ -606,12 +608,12 @@ Tests: 10 passed, 10 total Assertions: 25 passed, 25 total All tests passed +Time taken: 1.60s Slowest tests: 1.20s test_slow_database_query (tests/integration_test.sh) 340ms test_http_client_timeout (tests/http_test.sh) 12ms test_parse_config (tests/unit/config_test.sh) -Time taken: 1.60s ``` ```bash [Custom count] BASHUNIT_PROFILE_COUNT=3 bashunit test tests/ --profile @@ -667,8 +669,11 @@ bashunit test tests/ --retry 2 ```[Output] ✓ Passed: A flaky test (retry 1/2) -Tests: 1 passed, 1 total +Tests: 1 passed, 1 flaky, 1 total ``` + +A test that recovered on retry is counted as [flaky](#flaky-tests) as well as passed, so +the exit code stays `0` unless `--fail-on-flaky` is set. ::: It can also be set via the `BASHUNIT_RETRY` environment variable (see @@ -1111,7 +1116,7 @@ This is useful for: bashunit test tests/ --no-progress ``` ```[Output] -bashunit - 0.34.1 | Tests: 10 +bashunit | Tests: 10 Tests: 10 passed, 10 total Assertions: 25 passed, 25 total @@ -1208,13 +1213,18 @@ bashunit test tests/ --coverage --coverage-paths src/,lib/ --coverage-min 80 |---------------------------------|-----------------------------------------------------------------------------| | `--coverage` | Enable coverage tracking | | `--coverage-paths ` | Comma-separated paths to track (default: auto-discover from test files) | -| `--coverage-exclude ` | Comma-separated patterns to exclude (default: `tests/*,vendor/*,*_test.sh`) | +| `--coverage-exclude ` | Comma-separated patterns to exclude (default: `tests/*,vendor/*,*_test.sh,*Test.sh`) | | `--coverage-report [file]` | LCOV output file path (default: `coverage/lcov.info`) | | `--coverage-report-html [dir]` | Generate HTML report (default: `coverage/html`) | | `--coverage-min ` | Minimum coverage percentage; fails if below | | `--coverage-diff ` | Report only the lines changed since `` | | `--no-coverage-report` | Show console report only, don't generate LCOV file | +Both `--coverage-report` and `--coverage-report-html` take an optional value and fall back +to their default path when the next argument is another flag or absent. A value cannot be +told apart from a test path, so write the path before them: `bashunit tests/ --coverage-report`. + + ::: tip Coverage works with parallel execution (`-p`). Each worker tracks coverage independently, and results are aggregated before reporting. ::: @@ -1223,60 +1233,31 @@ Coverage works with parallel execution (`-p`). Each worker tracks coverage indep > `bashunit test --coverage --coverage-diff ` -Answers the question a pull request actually asks — *are the lines I touched -covered?* — instead of reporting a whole-file percentage that moves for reasons -unrelated to the change under review. +Restrict the console report to the lines changed since a base ref, so a pull request is +judged on the code it touched. See [Coverage > Diff coverage](/coverage#diff-coverage). -```bash -bashunit test tests/ --coverage --coverage-diff main -``` +## assert -``` -Diff Coverage (vs main) ---------------- -src/parser.sh 7/ 9 lines ( 77%) ---------------- -Total: 7/9 (77%) -``` +> `bashunit assert [args...]` +> +> `bashunit assert "" [ ...]` -Only lines **added or modified** since the ref are counted, from three sources -merged together: commits since the merge base, staged and unstaged edits, and -untracked files (counted in full). A pure deletion contributes nothing — there -is no line left to hold an opinion about — and changed lines that are not -executable (comments, `fi`, blank) are ignored, so a comment-only commit is not -penalised. +Run assertions without creating a test file. The function name works with or without the +`assert_` prefix. -The base ref is **required**. It is not defaulted, because an optional value -would make `--coverage-diff tests/` swallow the path as a ref. - -**With `--coverage-min`, the gate follows the report:** the threshold applies to -the diff percentage, so a change that fully covers itself passes even inside a -poorly covered file. - -```bash -bashunit test tests/ --coverage --coverage-diff origin/main --coverage-min 90 +::: code-group +```bash [Single assertion] +bashunit assert equals "foo" "foo" +bashunit assert exit_code 0 "echo 'success'" ``` - -A change with no executable lines scores **100%**, not 0% — otherwise a -docs-only commit would fail the gate. - -`--coverage-diff` restricts the **console report only**. LCOV and HTML stay -whole-file, because their consumers (`genhtml`, Codecov) do their own diffing -and expect complete records. - -::: warning Shallow clones -This needs `git` and a ref that resolves locally. CI checkouts are often shallow -and have no base ref, which would otherwise report "no changed lines" and pass a -threshold while measuring nothing — so an unresolvable ref is a hard error -instead. Fetch it first: - -```yaml -- uses: actions/checkout@v4 - with: - fetch-depth: 0 +```bash [Several assertions on one command] +bashunit assert "./my_script.sh" exit_code "0" contains "success" not_contains "error" ``` ::: +See [Standalone](/standalone) for the full story. `bashunit test --assert` is the +deprecated form of the first mode. + ## bench > `bashunit bench [path] [options]` @@ -1307,6 +1288,8 @@ bashunit bench --filter "parse" | `-vvv, --verbose` | Show execution details | | `--skip-env-file` | Skip `.env` loading, use shell environment only | | `-l, --login` | Run in login shell context | +| `--no-color` | Disable colored output (honors `NO_COLOR`) | +| `-h, --help` | Show the bench help | ## watch @@ -1314,10 +1297,13 @@ bashunit bench --filter "parse" Dedicated watch subcommand that uses **OS file-event notifications** (no polling) to re-run tests as soon as a `.sh` file changes. Any option accepted -by `bashunit test` is also accepted here. +by `bashunit test` is also accepted here, **but put the path first**: apart from +`-f/--filter`, an option's value is otherwise taken as the watch path, so +`bashunit watch --tag slow` watches a directory named `slow`. Write +`bashunit watch tests/ --tag slow`. When neither `inotifywait` nor `fswatch` is installed, it no longer fails: -it falls back to a **pure-shell polling loop** and prints a one-line notice. +it falls back to a **pure-shell polling loop** and prints a short notice. Polling checks every `BASHUNIT_WATCH_INTERVAL` seconds (default `2`) using `find -newer`, so it detects created and modified `.sh` files; deleted files are not detected on the fallback path. Install one of the tools above for @@ -1344,8 +1330,9 @@ bashunit watch tests/ --simple - **macOS:** `fswatch` (`brew install fswatch`) Without either tool, bashunit degrades to polling (see above) instead of -failing. The portable [`-w/--watch`](#watch-mode) flag on `bashunit test` -also uses polling. +failing. The portable [`-w/--watch`](#watch-mode) flag on `bashunit test` also polls, but +on a fixed 1-second loop: `BASHUNIT_WATCH_INTERVAL` applies to this subcommand's fallback +only. ::: ## doc @@ -1405,13 +1392,21 @@ bashunit init bashunit init spec ``` ```[Output] +> Created tests/bootstrap.sh +> Created tests/example_test.sh +> Created .github/workflows/tests.yml > bashunit initialized in tests ``` ::: Creates: -- `bootstrap.sh` - Setup file for test configuration -- `example_test.sh` - Sample test file to get started +- `tests/bootstrap.sh` - Setup file for test configuration +- `tests/example_test.sh` - Sample test file to get started +- `.github/workflows/tests.yml` - CI workflow using the official action +- `.env` with `BASHUNIT_BOOTSTRAP=tests/bootstrap.sh` + +An existing `BASHUNIT_BOOTSTRAP=` line in `.env` is commented out first, so the new value +takes effect. ## learn @@ -1463,7 +1458,7 @@ bashunit upgrade ``` ```[Output] > Upgrading bashunit to latest version -> bashunit upgraded successfully to latest version 0.34.1 +> bashunit upgraded successfully to latest version ``` ::: @@ -1549,7 +1544,9 @@ Without this, `--parralel` ran the suite **sequentially** and still exited `0`, ### Invalid values `--jobs`, `--retry`, `--test-timeout`, `--coverage-min` and `--seed` require a -non-negative integer; `--output` accepts only `tap`; `--shard` requires `/`: +non-negative integer; `--repeat` requires an integer of at least `1`; `--output` accepts +only `tap`; `--shard` requires `/`; and `--gha-annotations`, `--order-by` and +`--list-format` accept only their listed modes: ```bash bashunit --jobs abc tests/ @@ -1592,3 +1589,4 @@ risky also exits `0`. Add [`--fail-on-risky`](#test-options) or read the counts +- [Standalone](/standalone) — run assertions without a test file diff --git a/docs/configuration.md b/docs/configuration.md index 38fbcca1..4012a886 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -7,9 +7,12 @@ description: "Configure bashunit with environment variables and config files to Environment variables and config files control **bashunit** behavior across your project. It serves to configure the behavior of bashunit in your project. -You need to create a `.env` file in the root directory, -but you can give it another name if you pass it as an argument to the command with -`--env` [option](/command-line#environment-bootstrap). +`.env` and `.bashunitrc` are project files, always read from the working directory. + +`-e, --env` (alias `--boot`) loads an **additional** bootstrap file whose assignments +override them; it does not rename `.env`. See +[the option](/command-line#environment-bootstrap). `--skip-env-file` is the only way to +stop `.env` and `.bashunitrc` from being read at all. ## Config file (.bashunitrc) @@ -27,10 +30,16 @@ It is meant for committing sensible project defaults. Precedence, from highest to lowest: 1. CLI flags (e.g. `--simple`) -2. `.env` entries that have a value -3. Environment variables -4. `.bashunitrc` -5. Built-in defaults +2. The file given to `-e, --env, --boot` +3. `.env` entries that have a value +4. Environment variables +5. `.bashunitrc` +6. Built-in defaults + +The `--env` file is sourced during flag parsing, so it overrides `.env`, `.bashunitrc` and +the ambient environment, and it loses only to flags written **after** it on the command +line: `bashunit --simple --env custom.env` uses the file's value, `bashunit --env custom.env --simple` +uses the flag. Unlike `.env`, an empty entry in that file does wipe a value. An entry left **empty** in `.env` means "not configured here" and does not override the environment, so `BASHUNIT_OUTPUT_FORMAT=tap ./bashunit` keeps @@ -41,11 +50,17 @@ committed project config is for. `.bashunitrc` only fills values that are not already set, so anything above it always wins. `--skip-env-file` skips both `.env` and `.bashunitrc`. +The two files are read differently, which is why they behave differently: `.env` is +**sourced** as a shell script under `allexport`, so it can hold arbitrary shell and an +empty entry is unconditional (hence the preservation rule above), while `.bashunitrc` is +parsed as literal `KEY=value` lines and only fills names that are not already set. + ## Default path > `BASHUNIT_DEFAULT_PATH=directory|file` -Specifies the `directory` or `file` containing the tests to be run. `empty` by default. +Specifies the `directory` or `file` containing the tests to be run. `tests` by default, +so a run with no path argument searches `tests/` for files ending in `test.sh`. If a directory is specified, it will execute tests within files ending in `test.sh`. When running benchmarks (`bashunit bench`), the same path is used to search for files ending in `bench.sh`. @@ -71,9 +86,12 @@ BASHUNIT_DEFAULT_PATH=tests/**/*_test.sh Enables simplified output to the console. `false` by default. -Verbose is the default output, but it can be overridden by the environment configuration. +Detailed output is the default, but it can be overridden by the environment configuration. -Similar as using `-s|--simple | -vvv|--detailed` option on the [command line](/command-line#output-style). +Similar as using `-s|--simple` / `--detailed` on the [command line](/command-line#output-style). + +This is a different setting from [`BASHUNIT_VERBOSE`](#verbose): `-vvv|--verbose` adds the +execution-details block and does not change the result style. ::: code-group ```bash [Simple output] @@ -85,7 +103,7 @@ BASHUNIT_SIMPLE_OUTPUT=true ::: ::: code-group -```[Verbose output] +```[Detailed output] Running tests/functional/logic_test.sh ✓ Passed: Other way of using the exit code ✓ Passed: Should validate a non ok exit code @@ -101,7 +119,14 @@ BASHUNIT_SIMPLE_OUTPUT=false > `BASHUNIT_PARALLEL_RUN=true|false` -Runs the tests in child processes with randomized execution, which may improve overall testing speed, especially for larger test suites. +Runs the tests in child processes, one worker per test, which may improve overall testing +speed, especially for larger test suites. `false` by default. + +Dispatch keeps definition order, but **completion** order is nondeterministic, so tests +must not depend on each other. Parallel never shuffles: shuffling is opt-in through +[`BASHUNIT_RANDOM_ORDER`](#random-order) or `BASHUNIT_ORDER_BY=random`. Cap the concurrency +with [`BASHUNIT_PARALLEL_JOBS`](#parallel-jobs), and use `--no-parallel` to opt out of a +configured parallel run. ::: warning Parallel execution is supported on **macOS**, **Ubuntu**, **Alpine**, and @@ -179,6 +204,25 @@ BASHUNIT_RETRY=0 ``` ::: +## Repeat + +> `BASHUNIT_REPEAT=` + +Run each selected test `n` times so flakiness surfaces before CI hits it. `1` by default. +The test is reported once with the aggregate outcome, and a failure names the iteration it +happened on. Repeat wraps [`BASHUNIT_RETRY`](#retry), not the other way round. + +Similar as using `--repeat` option on the [command line](/command-line#repeat). + +## Fail on flaky + +> `BASHUNIT_FAIL_ON_FLAKY=true|false` + +Turn a run red when a test passed only after a retry. `false` by default, so a flaky test +stays inside the pass total and the exit code is unchanged. + +Similar as using `--fail-on-flaky` option on the [command line](/command-line#fail-on-flaky). + ## Random order > `BASHUNIT_RANDOM_ORDER=true|false` and `BASHUNIT_SEED=` @@ -201,6 +245,51 @@ BASHUNIT_RANDOM_ORDER=false ``` ::: +## Execution order + +> `BASHUNIT_ORDER_BY=defined|defects|random` + +Execution order. `defined` by default (definition order). `defects` runs the last run's +failures first and still runs the whole suite. `random` is the same mode as +[`BASHUNIT_RANDOM_ORDER=true`](#random-order). + +Similar as using `--order-by` option on the [command line](/command-line#order-by). + +## Exclude filter + +> `BASHUNIT_EXCLUDE_FILTER=name` + +Skip tests whose name matches. Empty by default. It wins over a `--filter` match. + +Similar as using `--exclude-filter` option on the [command line](/command-line#exclude-filter). + +## Changed files only + +> `BASHUNIT_CHANGED=true|false` and `BASHUNIT_CHANGED_REF=` + +Run only the test files git reports as changed. `false` by default. +`BASHUNIT_CHANGED_REF` is empty by default, which means `origin/HEAD`, then `HEAD`. + +Similar as using `--changed` option on the [command line](/command-line#changed). + +## Shard + +> `BASHUNIT_SHARD_INDEX=` and `BASHUNIT_SHARD_TOTAL=` + +Run shard `i` of `n` to split the suite across runners. Both are empty (disabled) by +default, and sharding needs both. + +Similar as using `--shard` option on the [command line](/command-line#shard). + +## List tests + +> `BASHUNIT_LIST_TESTS=true|false` and `BASHUNIT_LIST_FORMAT=text|json` + +Print the tests a run would execute and exit without running them. `false` and `text` by +default. + +Similar as using `--list` / `--list-format` options on the [command line](/command-line#list). + ## Snapshot update > `BASHUNIT_SNAPSHOT_UPDATE=true|false` @@ -243,6 +332,15 @@ BASHUNIT_SNAPSHOT_CREATE=true ``` ::: +## Snapshot report unused + +> `BASHUNIT_SNAPSHOT_REPORT_UNUSED=true|false` + +List the snapshot files no test resolved. `false` by default. Full runs only, and it deletes +nothing. + +Similar as using `--snapshot-report-unused` option on the [command line](/command-line#snapshot-report-unused). + ## Rerun failed > `BASHUNIT_RERUN_FAILED=true|false` @@ -507,6 +605,25 @@ The report destination is checked before the suite runs — an unwritable path f immediately instead of after a passing run. See [Invalid input](/command-line#invalid-input). ::: +## Report Markdown + +> `BASHUNIT_REPORT_MD=file` + +Write a Markdown run summary: verdict, counts table, failures with their message, plus +coverage and slowest tests when those ran. Empty by default. Inside GitHub Actions it is +also appended to `$GITHUB_STEP_SUMMARY`, for the outermost run only. + +Similar as using `--report-md` option on the [command line](/command-line#report-md). + +## GitHub Actions annotations + +> `BASHUNIT_GHA_ANNOTATIONS=auto|always|never` + +Controls GitHub Actions annotations on stdout. `auto` by default: on inside GitHub Actions, +quiet everywhere else. Never emitted under `--output tap`. + +Similar as using `--gha-annotations` option on the [command line](/command-line#gha-annotations). + ## Bootstrap > `BASHUNIT_BOOTSTRAP=file` @@ -577,7 +694,7 @@ bashunit::log "I am tracing something..." bashunit::log "error" "an" "error" "message" bashunit::log "warning" "different log level messages!" ``` -```bash [Output: out.log] +```[Output: dev.log] 2024-10-03 21:27:23 [INFO]: I am tracing something... #tests/sample.sh:11 2024-10-03 21:27:23 [ERROR]: an error message #tests/sample.sh:27 2024-10-03 21:27:24 [WARNING]: different log level messages! #tests/sample.sh:21 @@ -589,6 +706,13 @@ quickly `tail -f` it while the tests run. > All internal messages emitted by bashunit are prefixed with `[INTERNAL]`. > You can toggle internal messages with `BASHUNIT_INTERNAL_LOG=true|false`. +## Bench mode + +> `BASHUNIT_BENCH_MODE=true|false` + +Set by the `bashunit bench` subcommand and not meant to be configured by hand. `false` by +default. + ## Verbose > `BASHUNIT_VERBOSE=bool` @@ -738,9 +862,12 @@ BASHUNIT_NO_DIFF=true ## Color output -> `NO_COLOR=1` +> `BASHUNIT_NO_COLOR=true|false` + +Disables ANSI color codes in output. `false` by default. -Disables ANSI color codes in output. Follows the [no-color.org](https://no-color.org) standard. +`NO_COLOR` is the honored external standard: a non-empty `NO_COLOR` forces +`BASHUNIT_NO_COLOR=true`. Follows [no-color.org](https://no-color.org). When set to any value, bashunit will output plain text without color formatting. @@ -872,18 +999,49 @@ BASHUNIT_COVERAGE_EXCLUDE=tests/*,vendor/*,*_test.sh,*_mock.sh Path for the LCOV format coverage report. `coverage/lcov.info` by default. -Set to empty string to disable file generation (console report only). +An empty entry means "not configured", so the default path still applies. To skip the file +and keep the console report only, use the `--no-coverage-report` flag. ::: code-group ```bash [.env] # Custom path BASHUNIT_COVERAGE_REPORT=reports/coverage.lcov - -# Disable file output -BASHUNIT_COVERAGE_REPORT= +``` +```bash [Console only] +bashunit tests/ --coverage --no-coverage-report ``` ::: +### Coverage report HTML + +> `BASHUNIT_COVERAGE_REPORT_HTML=dir` + +Directory for the browsable HTML coverage report. Empty by default (not generated). + +Similar as using `--coverage-report-html` option on the [command line](/command-line#coverage). + +### Coverage diff + +> `BASHUNIT_COVERAGE_DIFF=` + +Restrict the console coverage report to the lines changed since ``. Empty by default. +`BASHUNIT_COVERAGE_MIN` then gates on that diff percentage instead of the whole-file one. +LCOV and HTML stay whole-file. + +Similar as using `--coverage-diff` option on the [command line](/command-line#coverage). + +### Coverage detail blocks + +> `BASHUNIT_COVERAGE_SHOW_FUNCTIONS=true|false` +> +> `BASHUNIT_COVERAGE_SHOW_UNCOVERED=true|false` +> +> `BASHUNIT_COVERAGE_SHOW_LINE_HITS=true|false` + +Opt-in blocks appended to the console coverage report: a per-function table, the executable +lines never hit (as compressed ranges), and per-line execution counts. All `false` by +default. + ### Coverage engine > `BASHUNIT_COVERAGE_ENGINE=auto|xtrace|trap` @@ -925,9 +1083,9 @@ BASHUNIT_COVERAGE_MIN=80 Thresholds for color-coding the coverage output. Defaults: `50` and `80`. -- Below `THRESHOLD_LOW`: Red -- Between thresholds: Yellow -- Above `THRESHOLD_HIGH`: Green +- At or above `THRESHOLD_HIGH`: green +- At or above `THRESHOLD_LOW`: yellow +- Below `THRESHOLD_LOW`: red ::: code-group ```bash [.env] @@ -938,9 +1096,9 @@ BASHUNIT_COVERAGE_THRESHOLD_HIGH=90 ## Deprecations -Every setting also answers to an **unprefixed** name — `VERBOSE` as well as -`BASHUNIT_VERBOSE`. Those unprefixed aliases predate the `BASHUNIT_` prefix -introduced in 0.15.0 and are deprecated: the names are generic enough that an +Many settings also answer to an **unprefixed** name — `VERBOSE` as well as +`BASHUNIT_VERBOSE`. Those are the ones that predate the `BASHUNIT_` prefix introduced in +0.15.0, and they are deprecated: the names are generic enough that an unrelated tool exporting `COVERAGE=true` or `VERBOSE=true` silently reconfigures bashunit. Always use the prefixed name. @@ -950,6 +1108,10 @@ When a deprecated form is what supplied a value, bashunit says so on stderr: Deprecated: the unprefixed `VERBOSE`. Use `BASHUNIT_VERBOSE` instead. ``` +Settings added after the prefix ship **only** under `BASHUNIT_`, so an unprefixed form for +them is ignored rather than warned about. `--retry`, `--seed`, `--order-by`, `--list`, and +the snapshot and shard settings are all prefix-only. + The warning goes to stderr, so it never corrupts a report on stdout. Silence it with: diff --git a/docs/coverage.md b/docs/coverage.md index 663fc7e1..5b41012f 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -18,7 +18,7 @@ bashunit tests/ --coverage bashunit tests/ --coverage-paths src/ ``` ```bash [Output] -bashunit - 0.37.0 | Tests: 5 +bashunit | Tests: 5 ..... Tests: 5 passed, 5 total @@ -43,10 +43,17 @@ Coverage report written to: coverage/lcov.info bashunit records which lines ran, then classifies and reports them: 1. **Capture**: every executed line's file path and line number is recorded, either from a `DEBUG` trap or from `xtrace` — see [Tracing engine](#tracing-engine) -2. **Filtering**: only files matching your coverage paths (and not excluded) are tracked +2. **Filtering**: a recorded file is tracked only when it matches your coverage paths and no exclude pattern 3. **Aggregation**: after tests complete, hit data is aggregated 4. **Reporting**: each source line is classified as executable or not, and the executable ones are matched against the hits +::: warning Only executed files are reported +A file enters the report the first time one of its lines runs. A source file that no test +touched at all is **absent** from the report rather than shown at 0%, so the percentage is +measured over the files that ran, not over everything under `BASHUNIT_COVERAGE_PATHS`. +Tracked in [#1053](https://github.com/TypedDevs/bashunit/issues/1053). +::: + ::: tip Performance Coverage roughly doubles to quadruples wall-clock time, depending on Bash version and engine. Cost is split fairly evenly between capture and reporting, and the @@ -82,9 +89,10 @@ Warning: coverage engine 'xtrace' needs Bash 4.1+ (running 3.2); using 'trap'. | `--coverage` | Enable code coverage tracking | | `--coverage-paths ` | Comma-separated paths to track (default: auto-discover from test files) | | `--coverage-exclude ` | Comma-separated exclusion patterns | -| `--coverage-report ` | LCOV report output path (default: `coverage/lcov.info`) | +| `--coverage-report [file]` | LCOV report output path (default: `coverage/lcov.info`) | | `--coverage-report-html [dir]` | Generate HTML report (default: `coverage/html`) | | `--coverage-min ` | Minimum coverage threshold (fails if below) | +| `--coverage-diff ` | Report only the lines changed since ``, see [Diff coverage](#diff-coverage) | | `--no-coverage-report` | Disable LCOV file generation (console only) | ::: tip Auto-enable @@ -108,40 +116,23 @@ For most projects following standard naming conventions, you can simply run `bas ### Environment Variables -You can also configure coverage via [environment variables](/configuration) in your `.env` file: +Every coverage flag has a matching setting. The full list, with defaults, lives in +[Configuration > Coverage](/configuration#coverage). The short version: ```bash -# Enable coverage -BASHUNIT_COVERAGE=true - -# Paths to track (comma-separated) -BASHUNIT_COVERAGE_PATHS=src/,lib/ - -# Patterns to exclude (comma-separated) -BASHUNIT_COVERAGE_EXCLUDE=tests/*,vendor/*,*_test.sh - -# LCOV report output path +BASHUNIT_COVERAGE=true # enable tracking +BASHUNIT_COVERAGE_PATHS=src/,lib/ # paths to track BASHUNIT_COVERAGE_REPORT=coverage/lcov.info - -# HTML report output directory (generates line-by-line coverage view) BASHUNIT_COVERAGE_REPORT_HTML=coverage/html - -# Minimum coverage percentage (optional) -BASHUNIT_COVERAGE_MIN=80 - -# Color thresholds for console output -BASHUNIT_COVERAGE_THRESHOLD_LOW=50 # Red below this -BASHUNIT_COVERAGE_THRESHOLD_HIGH=80 # Green above this, yellow between - -# Tracing engine: auto (default), xtrace or trap -BASHUNIT_COVERAGE_ENGINE=auto - -# Optional text-report blocks (off by default, opt-in for verbose runs) -BASHUNIT_COVERAGE_SHOW_FUNCTIONS=true # Print per-function coverage -BASHUNIT_COVERAGE_SHOW_UNCOVERED=true # Print missed line ranges per file -BASHUNIT_COVERAGE_SHOW_LINE_HITS=true # Print per-line execution counts (lineno:count) +BASHUNIT_COVERAGE_MIN=80 # fail below this percentage +BASHUNIT_COVERAGE_ENGINE=auto # auto, xtrace or trap ``` +Three opt-in blocks add detail to the console report: +`BASHUNIT_COVERAGE_SHOW_FUNCTIONS` (per-function table), +`BASHUNIT_COVERAGE_SHOW_UNCOVERED` (missed line ranges), +`BASHUNIT_COVERAGE_SHOW_LINE_HITS` (per-line execution counts). + ### Tracing engine Coverage can capture executed lines two ways, selected with @@ -560,6 +551,64 @@ Note the workflow excludes the engine's own meta-tests (`tests/unit/coverage_*_t from the measured run: executing them under `--coverage` double-instruments `src/coverage.sh` and corrupts their assertions. +## Diff coverage + +> `bashunit test --coverage --coverage-diff ` + +Answers the question a pull request actually asks — *are the lines I touched +covered?* — instead of reporting a whole-file percentage that moves for reasons +unrelated to the change under review. + +```bash +bashunit test tests/ --coverage --coverage-diff main +``` + +``` +Diff Coverage (vs main) +--------------- +src/parser.sh 7/ 9 lines ( 77%) +--------------- +Total: 7/9 (77%) +``` + +Only lines **added or modified** since the ref are counted, from three sources +merged together: commits since the merge base, staged and unstaged edits, and +untracked files (counted in full). A pure deletion contributes nothing — there +is no line left to hold an opinion about — and changed lines that are not +executable (comments, `fi`, blank) are ignored, so a comment-only commit is not +penalised. + +The base ref is **required**. It is not defaulted, because an optional value +would make `--coverage-diff tests/` swallow the path as a ref. + +**With `--coverage-min`, the gate follows the report:** the threshold applies to +the diff percentage, so a change that fully covers itself passes even inside a +poorly covered file. + +```bash +bashunit test tests/ --coverage --coverage-diff origin/main --coverage-min 90 +``` + +A change with no executable lines scores **100%**, not 0% — otherwise a +docs-only commit would fail the gate. + +`--coverage-diff` restricts the **console report only**. LCOV and HTML stay +whole-file, because their consumers (`genhtml`, Codecov) do their own diffing +and expect complete records. + +::: warning Shallow clones +This needs `git` and a ref that resolves locally. CI checkouts are often shallow +and have no base ref, which would otherwise report "no changed lines" and pass a +threshold while measuring nothing — so an unresolvable ref is a hard error +instead. Fetch it first: + +```yaml +- uses: actions/checkout@v4 + with: + fetch-depth: 0 +``` +::: + ## Related - [Command-Line](/command-line) — full reference for CLI flags and options diff --git a/docs/custom-asserts.md b/docs/custom-asserts.md index 62a24b10..0885e565 100644 --- a/docs/custom-asserts.md +++ b/docs/custom-asserts.md @@ -12,6 +12,10 @@ Check the internal functional tests: `tests/functional/custom_asserts_test.sh` ( ::: info Assertion behavior When using the bashunit facade, assertions automatically respect the guard behavior: if a previous assertion in the same test already failed, subsequent assertions are skipped. This matches popular testing libraries default behavior. + +This is the default. `-R`/`--run-all` (`BASHUNIT_STOP_ON_ASSERTION_FAILURE=false`) disables +the guard, so every assertion inside your custom assertion runs and counts even after an +earlier failure. ::: ::: info Test name detection @@ -31,7 +35,9 @@ Runs `cmd` and marks the assertion passed or failed accordingly, in a single cal | `actual` | The actual value received | | `cmd [args...]` | The command deciding the verdict: exit `0` passes, anything else fails | -Returns `0` when the command succeeds and `1` when it fails, so it can be chained. +Returns `0` when the command succeeds and `1` when it fails, so it can be chained. Leaving +it as the last statement of your custom assertion is fine: a failed assertion is reported +once, as a failure. The command is invoked directly, without `eval`, so arguments keep their word boundaries and nothing is re-parsed by the shell. @@ -65,6 +71,12 @@ Marks the current assertion as failed and prints a failure message. Marks the current assertion as passed. Call this when your custom assertion succeeds. +### fail +> `bashunit::fail ` + +Marks the current assertion as failed and prints `Message: ''` instead of the +Expected/but-got block. Use it when there is no meaningful expected value. + ## Examples ### One-call assertion @@ -123,7 +135,7 @@ function assert_http_success() { function assert_foo() { local actual="$1" - if [[ "foo" != "$actual" ]]; then + if [ "foo" != "$actual" ]; then bashunit::assertion_failed "foo" "$actual" return fi @@ -136,7 +148,7 @@ function test_value_is_foo() { } function test_value_is_not_foo() { - assert_foo "bar" # Fails with: "Failed: Value is not foo" + assert_foo "bar" # Fails with: "Expected 'foo' but got 'bar'" } ``` @@ -204,10 +216,7 @@ Assertions: 1 passed, 1 failed, 2 total The failure now reads `Expected 'a 2xx status' but got '500'`, labelled with the test that called it. -| Parameter | Description | -|-----------|-------------| -| `label` | What the assertion expects, shown in the failure block. Omit it to keep reporting the innermost failure message | -| `actual` | The actual value shown against that label | +Parameters: see [assert_once](#assert-once). Notes: @@ -228,7 +237,7 @@ Notes: function assert_positive_number() { local actual="$1" - if [[ "$actual" -le 0 ]]; then + if [ "$actual" -le 0 ]; then bashunit::assertion_failed "positive number" "$actual" "got" return fi @@ -310,15 +319,20 @@ assertions, it appends them too, rendering the comment block above each one: ```bash ./bashunit doc --boot tests/bootstrap.sh # built-ins, then a "Custom assertions" section ./bashunit doc --custom --boot tests/bootstrap.sh # only your own -./bashunit doc --custom http # ...narrowed by a filter +./bashunit doc --custom --boot tests/bootstrap.sh http # ...narrowed by a filter ``` ``` ## assert_http_success -------------- Asserts that the status code is a 2xx. +Arguments: $1 - the status code ``` +The whole comment run above the function is printed, so a two-line docstring renders as two +lines. A function with no comment block prints its heading and the divider with nothing +under them. + With `BASHUNIT_BOOTSTRAP` set, the `--boot` flag can be omitted. A bootstrap is required either way: it is the only point at which your assertions are guaranteed loaded, which is another reason to prefer it over sourcing from @@ -344,7 +358,7 @@ function assert_http_success() { 2. **Always return after failure**: when writing the long form by hand, call `return` after `bashunit::assertion_failed` or `bashunit::fail` to stop execution of your custom assertion. -3. **Always mark success**: Call `bashunit::assertion_passed` or `state::add_assertions_passed` when your assertion succeeds. +3. **Always mark success**: call `bashunit::assertion_passed` when your assertion succeeds. 4. **Use descriptive names**: Name your custom assertions clearly, e.g., `assert_valid_email`, `assert_file_contains_header`. diff --git a/docs/data-providers.md b/docs/data-providers.md index 919968af..96319b2f 100644 --- a/docs/data-providers.md +++ b/docs/data-providers.md @@ -25,12 +25,49 @@ function test_my_test_case() { ``` ::: +The annotation must sit within **two lines** of the function definition. A `# @tag` line or +a one-line docstring in between is fine; anything longer and bashunit stops seeing the +annotation, and the test runs once with no arguments instead of reporting an error. + ## Implementing a data provider A data provider function contains one or more `bashunit::data_set` lines. Each `bashunit::data_set` results in a separate run of the test function with the individual `bashunit::data_set` arguments being passed to it as positional arguments (`$1`, `$2`, ...). Each run is treated as a separate test, so it can pass or fail independently. Plus, [set_up](/test-files#set-up-function) and [tear_down](/test-files#tear-down-function) are called before and after each run. This reduces code repetition and helps create related tests more efficiently. +Under `--parallel` every row is dispatched as its own concurrent job, so rows sharing a +file, a temp path or an environment variable will race. Serialise them with the +`# bashunit: no-parallel-tests` directive at the top of the file (see +[Parallel](/command-line#parallel)), or give each row its own scratch path with +`bashunit::temp_file`. + +The annotation must be in the test file, but the provider function only has to be defined +by the time the run starts. Put shared providers in your bootstrap file (`--boot` or +`BASHUNIT_BOOTSTRAP`) and reference them by name from any test file: + +::: code-group +```bash [tests/bootstrap.sh] +function provider_supported_shells() { + bashunit::data_set "bash" + bashunit::data_set "zsh" +} +``` +```bash [tests/any_test.sh] +# @data_provider provider_supported_shells +function test_shell_is_available() { + assert_command_available "$1" +} +``` +::: + +::: warning A provider with no rows makes its test disappear +A provider that does not exist, or that emits no `bashunit::data_set` line, makes its test +run **zero** times, and bashunit does not error: the header still counts the test, nothing +runs, and the suite stays green. Compare the header count with the reported total (`Tests: 2` +in the header but `1 total` at the bottom means a provider produced no rows), or check the +selection with `./bashunit --list `. +::: + A data provider function is implemented as follows: ::: code-group @@ -151,6 +188,33 @@ Running example_test.sh ``` ::: +## Failing rows + +A failing row's result line is labelled with the test name only: the arguments are **not** +appended the way they are on a passing row. + +```[Output] +✓ Passed: Directory exists ('/usr') +✓ Passed: Directory exists ('/etc') +✗ Failed: Directory exists + Expected '/nope' + to exist but 'do not exist' + at example_test.sh:5 +``` + +Identify the row from the `Expected` value, or use `::1::` interpolation in the test title, +which does put the arguments in the name. + +## Combining with other options + +| Option | Effect on a provider | +|--------|----------------------| +| `--repeat ` | Every row runs n times; each row still reports one line | +| `--retry ` | Only the failing row is retried | +| `# @tag` / `--tag` | Tags belong to the test function, so they select all its rows | +| `--filter` | Matches the **function name**, not the interpolated title: `--filter directory_exists`, not `--filter "Directory exists"` | +| `--list` | Lists the function once; the row count is a property of the run (see [Command line](/command-line#list)) | + ## Related - [Test files](/test-files) — `set_up` and `tear_down` lifecycle hooks diff --git a/docs/index.md b/docs/index.md index 802a4e03..808bb8e9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -25,14 +25,14 @@ features: - icon: src: /flexible.svg title: Flexible - details: Robust assertions for comparing, matching, and validating results, ensuring thorough testing of your codebase. + details: 84 assertions plus mocks, spies, data providers and snapshots, for comparing, matching and validating anything your scripts produce. - icon: src: /accessible.svg - title: Accessible - details: An intuitive API and clear documentation for a smooth developer experience, reducing testing complexity. + title: Fast and CI-ready + details: Run in parallel or shard the suite across runners, run only what changed, measure coverage, and publish JUnit, TAP, JSON, HTML or Markdown reports. - icon: src: /updated.svg - title: Updated + title: Community details: A vibrant GitHub community for support, collaboration, and continuous library enhancement. Join forces with like-minded developers. - icon: src: /multiplatform.svg diff --git a/docs/installation.md b/docs/installation.md index 9dc1ca00..607ae60c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -8,7 +8,18 @@ description: "Install bashunit via install.sh, npm, Brew, MacPorts or bashdep: a ## Requirements -bashunit requires **Bash 3.0** or newer. On Windows use [WSL](https://learn.microsoft.com/windows/wsl/install). +bashunit requires **Bash 3.0** or newer. + +On Windows, install [WSL](https://learn.microsoft.com/windows/wsl/install) (`wsl --install` +from an elevated PowerShell, then reboot) and run every command below inside the WSL shell. + +Everything else is optional, but some features need it: + +| Tool | Needed for | Without it | +|------|------------|------------| +| `jq` | [JSON assertions](/assertions#assert-json-key-exists) | the test is **skipped**, not failed | +| `git` | `--changed`, `--coverage-diff`, `upgrade`, failure diffs | those flags error out; diffs print plain values | +| `inotifywait` (Linux) / `fswatch` (macOS) | [`watch`](/command-line#watch) | falls back to polling | ## install.sh @@ -19,26 +30,21 @@ There is a tool that will generate an executable with the whole library in a sin curl -s https://bashunit.com/install.sh | bash ``` -```bash [Windows] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: -curl -s https://bashunit.com/install.sh | bash -``` ::: This will create a file inside a lib folder, such as `lib/bashunit`. ::: tip Automatic checksum verification -`install.sh` verifies the download against the release `checksum` asset by default and -aborts on a mismatch, so a tampered or corrupted download never lands. Set -`BASHUNIT_VERIFY_CHECKSUM=false` to opt out (e.g. for old releases published before -checksum assets existed). The manual check below is only needed when you opt out. +`install.sh` verifies the download against the release `checksum` asset and aborts on a +mismatch, so a tampered or corrupted download never lands. There are three states: + +- **unset** (default): verifies, aborts on a mismatch, and only *warns and continues* when + the checksum asset or a sha256 tool is unavailable +- `BASHUNIT_VERIFY_CHECKSUM=true`: also aborts when verification is impossible +- `BASHUNIT_VERIFY_CHECKSUM=false`: skips verification entirely + +The GitHub Action always passes the variable, so `verify-checksum: 'true'` (its default) is +the strict mode. The manual check below is only needed when you skip verification. ::: #### Verify @@ -64,17 +70,6 @@ The installation script can receive arguments (in any order): curl -s https://bashunit.com/install.sh | bash -s [dir] [version] ``` -```bash [Windows] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: -curl -s https://bashunit.com/install.sh | bash -s [dir] [version] -``` ::: - `[dir]`: the destiny directory to save the executable bashunit; `lib` by default @@ -236,14 +231,7 @@ bashdep::install "${DEPENDENCIES[@]}" ``` ```bash-vue [Windows - install-dependencies.sh] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: +# Run this inside the WSL shell, see Requirements above. # Ensure bashdep is installed [ ! -f lib/bashdep ] && { @@ -290,10 +278,11 @@ jobs: # For an immutable pin use a commit SHA: TypedDevs/bashunit@ # {{ pkg.version }} - uses: TypedDevs/bashunit@v0 with: - version: '{{ pkg.version }}' # or "latest" (default) + version: '{{ pkg.version }}' # omit for the version pinned at this ref directory: lib # optional, "lib" by default add-to-path: 'true' # optional, "true" by default verify-checksum: 'true' # optional, "true" by default + annotations: auto # optional, "auto" by default ("never" to turn off) # add-to-path puts the binary on $PATH, so just call "bashunit": - run: bashunit tests ``` @@ -343,7 +332,7 @@ jobs: ``` ::: -**Inputs:** `version` (default `latest`), `directory` (default `lib`), `add-to-path` (default `true`), `verify-checksum` (default `true`), `args` (default empty — when set, runs `bashunit ` after installing). +**Inputs:** `version` (default: the release pinned at this action ref, e.g. `0.45.0` on `@v0`; pass `latest` to always take the newest), `directory` (default `lib`), `add-to-path` (default `true`), `verify-checksum` (default `true`), `args` (default empty — when set, runs `bashunit ` after installing), `annotations` (default `auto` — GitHub Actions annotations for failing tests; `never` turns them off, `always` forces them). **Outputs:** `path` (binary path relative to the workspace), `version` (installed version). `verify-checksum` validates the downloaded binary against the release `checksum` @@ -387,9 +376,27 @@ Either way you get bashunit updates as routine pull requests — no manual re-pi `curl | bash` bumps to remember. Review the PR, let CI run, merge. ::: tip -See bashunit's own pipeline for a real example: https://github.com/TypedDevs/bashunit/blob/main/.github/workflows/tests.yml +See the action's own end-to-end test for a real example, covering `version`, `directory`, +`add-to-path` and `args`: +https://github.com/TypedDevs/bashunit/blob/main/.github/workflows/test-action.yml + +bashunit's `tests.yml` runs the in-repo entrypoint instead, so it is not an install example. ::: +## Updating + +For the `install.sh`, GitHub Action and bashdep routes, upgrade the binary in place: + +```bash +./lib/bashunit upgrade +``` + +It downloads the newest release over the same file, and prints +`> You are already on latest version` when there is nothing to do. + +Package-manager installs update through their own manager instead: `brew upgrade bashunit`, +`sudo port upgrade bashunit`, `npm install --save-dev bashunit@latest`. + ## Shell completion bashunit ships tab-completion scripts for bash and zsh under diff --git a/docs/quickstart.md b/docs/quickstart.md index f1806e6e..8bc11ce0 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -30,17 +30,6 @@ npm install -g bashunit bashunit tests/ ``` -```bash [Windows] -# IMPORTANT: You need WSL (Windows Subsystem for Linux) to run bashunit -# -# Step 1: Install WSL if you haven't already -# - Open PowerShell as Administrator -# - Run: wsl --install -# - Restart your computer -# -# Step 2: Open your WSL terminal and run: -curl -s https://bashunit.com/install.sh | bash -``` ::: The `install.sh` route creates `lib/bashunit`; the npm route exposes `bashunit` via `npx` or your global `PATH`. @@ -55,7 +44,15 @@ You can bootstrap a ready to use test suite with the `init` subcommand: ./lib/bashunit init tests ``` -This will create a `tests` directory containing a sample test and bootstrap file. +It creates, in the current directory: + +- `tests/bootstrap.sh` — sourced before your tests; put shared setup here +- `tests/example_test.sh` — a sample test +- `.github/workflows/tests.yml` — a CI workflow using the official action +- `.env` — with `BASHUNIT_BOOTSTRAP=tests/bootstrap.sh`, which is what makes the bootstrap load + +If `.env` already sets `BASHUNIT_BOOTSTRAP`, `init` comments that line out and appends the +new one, so check the diff before committing. Alternatively, create your tests manually: @@ -77,14 +74,14 @@ Alternatively, create your tests manually: 3. Finally, run the **bashunit** executable: ```bash - ./lib/bashunit ./tests + ./lib/bashunit tests ``` 4. If everything works correctly, you should see an output similar to the following: ```-vue bashunit - {{ pkg.version }} | Tests: 1 Running tests/example_test.sh - ✓ Passed: Bashunit is working 16ms + ✓ Passed: Bashunit is working Tests: 1 passed, 1 total Assertions: 1 passed, 1 total @@ -93,8 +90,22 @@ Alternatively, create your tests manually: Time taken: 90ms ``` + A per-test duration column appears when the platform has a cheap clock source; + `BASHUNIT_SHOW_EXECUTION_TIME` controls it. + 5. Now you can start testing the functionalities of your own Bash scripts. +## Running your suite + +With no path, bashunit runs `tests/` (`BASHUNIT_DEFAULT_PATH`): + +```bash +./lib/bashunit # run tests/ +./lib/bashunit --filter user # only tests whose name matches +./lib/bashunit --parallel tests/ # run files concurrently +./lib/bashunit --changed # only test files touched since origin/HEAD +``` + ## Learning bashunit interactively If you prefer hands-on learning, bashunit includes an interactive tutorial: @@ -114,6 +125,8 @@ Dive deeper into the documentation: - **[Data providers](/data-providers)** - Write parameterized tests efficiently - **[Snapshots](snapshots)** - Test complex output easily - **[Test files](/test-files)** - Understand test file structure and lifecycle hooks +- **[Command line](/command-line)** - Every flag, from `--filter` to `--shard` +- **[Configuration](/configuration)** - `.env` and `BASHUNIT_*` variables