[Discussion] googletest(gtest) or Catch2

[zhjwpku]

Both can be easily integrated into our CMake build. As for the popularity, googletest rank 1 (33%) according to the 2023 JetBrains C++ ecosystem survey [1] while Catch2 rank 2 (12%).

Catch2 also provides basic micro-benchmarking features, and simple BDD macros, but I'm not sure do we need these features in our project.

I have no previous experience using Catch2, so I want to hear from the community, please comment which UT framework do you think we should use, and the reason why you prefer it.

[1] https://www.jetbrains.com/lp/devecosystem-2023/cpp/#cpp_unittesting_two_years

[pitrou]

As I wrote on your PR, my main concern with GTest is compilation times, but I don't know if Catch2 would improve things on this axis.

One question I have about Catch2 is whether it's possible to have rich error reports, for example I would like REQUIRE(lhs == rhs) to display the values of lhs and rhs if the equality fails.

[lidavidm]

Catch2 does have rich errors, but I'm rather rattled by how it implements it:

The individual test assertions are written using the REQUIRE macro. It accepts a boolean expression, and uses expression templates to internally decompose it, so that it can be individually stringified on test failure.

[zhjwpku]

As I wrote on your PR, my main concern with GTest is compilation times, but I don't know if Catch2 would improve things on this axis.

One question I have about Catch2 is whether it's possible to have rich error reports, for example I would like REQUIRE(lhs == rhs) to display the values of lhs and rhs if the equality fails.

Here are the failing messages for both catch2 and gtest, I have no opinion which is better ;)

[gaborkaszab]

No strong opinions from me. I myself gravitate towards gtest just because I have experience on that, but could live with any of them.

[zhjwpku]

No strong opinions from me. I myself gravitate towards gtest just because I have experience on that, but could live with any of them.

Yeah, both test frameworks seems capable of meeting our project's needs. However, it would be most beneficial for us to choose the one that the majority of our development team is already familiar with.

I have no experience on Catch2, so I give my vote to googletest.

[pitrou]

Arrow C++ and Parquet C++ developers are certainly familiar with GTest too.

[wgtmac]

I don't have experience with Catch2 either. But I don't object to adopt it if it brings good stuff.

[wgtmac]

Should we move forward? I don't see any objection to googletest and #13 looks good now.

cc @Fokko @Xuanwo

[zhjwpku]

In this discussion regarding the choice between Googletest and Catch2 for our testing framework, most of the developers have experience with Googletest, and while they are open to considering Catch2, they see no immediate advantage, as no one on the team has experience with it. Given this context, the decision to adopt Googletest was made based on the familiarity and comfort level of the majority of the team.

We can switch to Catch2 with one shot if it proves to be a better fit for our needs in the future.

As #13 has already been merged, I will close this issue, thanks for all the inputs.