Bash Testing: what are you using?

[meleu]

I like BATS, it's battle tested and is around for more than a decade. I just don't like the amount of setup required to start testing (and more setup to get nice assertions). I mean, it's easy for me, but intimidating for newcomers, therefore hard for us "testing evangelists" to convince people to start writing tests.

From time to time I feel seduced by the simplicity that bashunit promises, but as soon as I try it I stumble into something really simple that doesn't give me the sense of safety I expect from a testing framework (example 1, example 2).

What about you guys? What are you using for bash testing?

[DannyBen]

I love testing... for me, coming from Ruby that has the gold standard rspec and the more niche (but excellent) cucumber, anything that any other language has to offer falls short of my expectations.

I love approval-based testing. I am using it both for bash (approvals.bash) and for ruby (rspec_approvals), and it served me well for years.

However, recently I decided that I need a more "unit testing oriented" test coverage, and started switching to bats. These are my first two repos that test with bats: ssi and alf.

I think the problem in bash testing is not the framework, but the bash language itself. Stubbing and mocking in bash is awkward, which makes isolated unit testing harder to implement.

All in all, I feel that bats is doing a good job at providing an easy to use CLI, easy to read output, and simple framework for defining test cases.

I am also going to check ShellSpec which seems like a combination of gherkin and rspec for bash. Could be interesting.

[pcrockett]

Bats here too. Except I only use the core CLI. I don't use any of the plugins. They don't behave the way I expect, and as you mentioned, they require some setup.

Instead, I have some version of this script copy / pasted into my various repos. Here's an example of what a unit test file looks like for me: https://github.com/pcrockett/gramps/blob/main/tests/encrypt.bats

[meleu]

And once again, I got disappointed with bashunit... 😔

Lesson learned: sticking with BATS is a good way to prevent wasting time.

[DannyBen]

The thing is that after you see how beautiful tests can be in Ruby, everything else looks like a mess. At least BATS does not try to do too much.

[meleu]

By the way, I'm surprised you didn't mention minitest in your first reply from last week.

I like minitest more than RSpec, because it's "just Ruby", with no new DSL to learn. But I recognize that most jobs out there requires RSpec.

[DannyBen]

Yes, I started with minitest when I first learned Ruby, and refused to move to rspec for a long time. But after I did, I truly understood the power of Ruby, and this one of a kind library. Minitest is similar to testing frameworks in other languages, while rspec syntax cannot be reproduced in any other language. It is the ultimate showcase of ruby metaprogramming.

[DannyBen]

Also, if we already mention the test-framework-legends - I was always a big fan of Cucumber / Gherkin in Ruby, and I wonder if it would be a worthwhile effort to create such a framework for Bash. Imagine this:

# test.featureGiven "I am in temp directory"When "I run 'touch somefile'"Then "the file 'somefile' should exist"And "the file 'some other file' should exist" # failure

# stepsdefs.sh
@Step "I am in temp directory"
mkdir -p ./tmp
chdir tmp
@Step "I run '*'"command
must_run "$command"
@Step "I try '*'"command
run "$command"
@Step "the file '*' should exist" file
[[ -f"$file" ]]
@Step "the exit code should be *" code
[[ "$LAST_EXIT_CODE"-eq"$code" ]]

Output:

✓ Given "I am in temp directory"
✓ When "I run 'touch somefile'"
✓ Then "the file 'somefile' should exist"
✗ Then "the file 'some other file' should exist"
Step exited with status 1

I already have a vibe-coded prototype.

[DannyBen]

So.... I could not resist.
Enter: Shellkin - Gherkin-style BDD Testing Framework for Bash

built with bashly of course, tested with bats and dogfoods itself with initial feature testing as well.

[zen0bit]

I like clitest
There's no syntax to learn

Simple test shell scripts straight from Markdown documents

[DannyBen]

yeah, clitest is a nice concept. As it so happens, some time ago I created a similar concept for Ruby - docspec.

This is a simple form of approval/snapshot testing, which are perfect for quick-to-write and easy-to-maintain tests, but for me, I quickly ran out of runway with these approaches when I reached a point where I needed function-level unit testing or stubbing/mocking capabilities.

[meleu]

[off-topic]

I like clitest

By the way @aureliojargas is an old Brazilian buddy from who I learned a lot of shell scripting and sed/RegEx tricks in the late 90s.

[DannyBen]

So I think this is a little story worth telling.
At first, I created Shellkin just to "scratch an itch" and to see if it can be done in a way that step definitions are easy to write.
Then I realized, it is quite stable and very usable.

1. Shellkin - the gherkin-style test framework inspired by this discussion - was created in bashly
   • It was created entirely by codex, using the bashly skill.
   • Its tests - also written by codex - are done with bats and shellkin itself for acceptance.
2. I then decided to try it "in the real world".
   • I decided to migrate the approval tests for rush - one of my more hard-to-test bash scripts.
   • So I first created a Shellkin skill, again using codex to do so.
   • Using this skill, I supervised codex while it built the entire acceptance test suite for rush.

All this was done in a fraction of the time it would have taken me without bashly and without codex, and most of the time I was away from my computer eating pizza and watching netflix...

The tests include stubbing commands (such as git), testing in quasi-isolated environments (without docker, which was the previous isolation method), highly human-readable feature definitions, easy to understand and maintain step definitions.

I am quite happy with the result.

[meleu]

All this was done in a fraction of the time it would have taken me without bashly and without codex, and most of the time I was away from my computer eating pizza and watching netflix...

This looks like a line taken from an AI-hyped youtube video that "changes everything". 😅

[DannyBen]

This looks like a line taken from an AI-hyped youtube video that "changes everything". 😅

Well.... it does change everything - the last time I wrote a line of code myself is a distant memory now.... 😈