ARROW-11319: [Rust] [DataFusion] Improve test comparisons to record batch, remove test::format_batch

[alamb]

The test::format_batch function does not have wide range of type support (e.g. it doesn't support dictionaries) and its output makes tests hard to read / update, in my opinion. This PR consolidates the datafusion tests to use arrow::util::pretty::pretty_format_batches both to reduce code duplication as well as increase type support

This PR removes the test::format_batch(&batch); function and replaces it with arrow::util::pretty::pretty_format_batches and some macros. It has no code changes.

This change the following benefits:

1. Better type support (I immediately can compare RecordBatches with Dictionary types in tests without having to update format_batch and ARROW-11289: [Rust][DataFusion] Implement GROUP BY support for Dictionary Encoded columns #9233 gets simpler)
2. Better readability and error reporting (at least I find the code and diffs easier to understand)
3. Easier test update / review: it is easier to update the diffs (you can copy/paste the test output into the source code) and to review them

This is a variant of a strategy that I been using with success in IOx source link and I wanted to contribute it back.

An example failure with this PR:

---- physical_plan::hash_join::tests::join_left_one stdout ----
thread 'physical_plan::hash_join::tests::join_left_one' panicked at 'assertion failed: `(left == right)`
left: `["+----+----+----+----+", "| a1 | b2 | c1 | c2 |", "+----+----+----+----+", "| 1 | 1 | 7 | 70 |", "| 2 | 2 | 8 | 80 |", "| 2 | 2 | 9 | 80 |", "+----+----+----+----+"]`,
right: `["+----+----+----+----+----+", "| a1 | b1 | c1 | a2 | c2 |", "+----+----+----+----+----+", "| 1 | 4 | 7 | 10 | 70 |", "| 2 | 5 | 8 | 20 | 80 |", "| 3 | 7 | 9 | | |", "+----+----+----+----+----+"]`:
expected:
[
"+----+----+----+----+",
"| a1 | b2 | c1 | c2 |",
"+----+----+----+----+",
"| 1 | 1 | 7 | 70 |",
"| 2 | 2 | 8 | 80 |",
"| 2 | 2 | 9 | 80 |",
"+----+----+----+----+",
]
actual:
[
"+----+----+----+----+----+",
"| a1 | b1 | c1 | a2 | c2 |",
"+----+----+----+----+----+",
"| 1 | 4 | 7 | 10 | 70 |",
"| 2 | 5 | 8 | 20 | 80 |",
"| 3 | 7 | 9 | | |",
"+----+----+----+----+----+",
]

You can copy/paste the output of actual directly into the test code for an update.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[alamb]

format batch is similar, but not the same, as pretty_print

The major difference is that it renders NULL values as "NULL" whereas the pretty printer renders them as an empty string ""

[alamb]

@jorgecarleitao@andygrove and @seddonm1 and @Dandandan -- what do you think of this approach to testing DataFusion output?

[andygrove]

Thanks @alamb I really like this and I like that it is now also testing the column names.

We may also want to consider moving the expected results out to files rather than having them in the code, especially the larger ones.

[Dandandan]

LGTM. Very useful @alamb 👍

[alamb]

Rebasing to resolve a logical conflict (a new test that was added)

[codecov-io]

Codecov Report

Merging #9264 (363e1f4) into master (13e2134) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #9264 +/- ##
==========================================
- Coverage 81.87% 81.84% -0.03% 
==========================================
Files 215 215 Lines 53097 52949 -148 ==========================================
- Hits 43471 43336 -135 + Misses 9626 9613 -13 

Impacted Files	Coverage Δ
rust/datafusion/src/execution/context.rs	88.25% <100.00%> (-1.11%)	⬇️
...ust/datafusion/src/physical_plan/hash_aggregate.rs	82.17% <100.00%> (-0.13%)	⬇️
rust/datafusion/src/physical_plan/hash_join.rs	83.40% <100.00%> (-0.48%)	⬇️
rust/datafusion/src/test/mod.rs	100.00% <100.00%> (+10.07%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13e2134...363e1f4. Read the comment docs.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-11319

[jorgecarleitao]

I was unable to review this in time, but it is a good idea. Thanks a lot @alamb .

I was unsure what happens when there is a string "NULL" on the result vs the slot being null, but that is something that we are already doing anyways with the current approach, so LGTM.