Skip to content

test: cover struct data columns in native shuffle - #5564

Merged
viirya merged 1 commit into
apache:mainfrom
viirya:test-native-shuffle-struct-coverage
Aug 31, 2026
Merged

test: cover struct data columns in native shuffle#5564
viirya merged 1 commit into
apache:mainfrom
viirya:test-native-shuffle-struct-coverage

Conversation

@viirya

@viirya viirya commented Aug 30, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #5563.

Rationale for this change

CometShuffleExchangeExec.supportedSerializableDataType allows struct data
columns in native shuffle, but CometNativeShuffleSuite never exercised one --
the only nested coverage was array<array<int>> and map<int, null>, and the
word struct did not appear in the suite at all. CometColumnarShuffleSuite
has extensive nested coverage, so the gap was specific to native shuffle.

Nested types in shuffle have been a real source of bugs (e.g. cd4d0e2, #5137,
on nested field nullability in ShuffleScanExec), so a behavior the type gate
explicitly claims to support deserves an end-to-end test rather than relying on
it not regressing.

What changes are included in this PR?

Test-only. Four tests added to CometNativeShuffleSuite:

  • native shuffle on struct data column -- a plain struct<int, string>
  • native shuffle on struct data column including nulls -- a null struct and a
    null field inside a struct
  • native shuffle on deeply nested data columns -- struct<array<int>, string>
    and array<struct<int, string>>, covering the recursive branches of the gate
  • native shuffle on struct data column with map field

All of them partition on a primitive column and carry the struct as a data
column, because supportedHashPartitioningDataType rejects nested types as a
hash key -- a struct can only reach native shuffle as data. The first test
carries a comment explaining this, so the arrangement is not mistaken for an
oversight.

No product code is changed.

How are these changes tested?

This PR is the tests. CometNativeShuffleSuite passes locally: 40 tests,
40 succeeded, 0 failed (36 pre-existing + 4 new).

I also mutation-tested them to confirm they are not vacuous: temporarily
changing the native gate's case StructType(...) to false makes exactly these
4 tests fail while the other 36 keep passing, so they really do assert that
struct data columns go through native shuffle, rather than passing because the
plan silently fell back to Spark.

`CometShuffleExchangeExec.supportedSerializableDataType` allows struct data
columns in native shuffle, but `CometNativeShuffleSuite` never exercised one:
the only nested coverage was `array<array<int>>` and `map<int, null>`, and the
word "struct" did not appear in the suite at all. The columnar suite has
extensive nested coverage, so the gap was specific to native shuffle.

Struct columns can only reach native shuffle as DATA columns, since
`supportedHashPartitioningDataType` rejects nested types as a hash key, so
these tests partition on a primitive and carry the struct along.

Adds four tests:
- a plain `struct<int, string>` data column
- a struct data column with a null struct and a null field inside a struct
- `struct<array<int>, string>` and `array<struct<int, string>>`, covering the
  recursive branches of the type gate
- a struct data column with a map field

Co-authored-by: Claude Code <noreply@anthropic.com>
@viirya
viirya force-pushed the test-native-shuffle-struct-coverage branch from cc7148d to 59670b1 Compare August 31, 2026 04:50
@viirya
viirya merged commit 199a910 into apache:main Aug 31, 2026
100 of 101 checks passed
@viirya
viirya deleted the test-native-shuffle-struct-coverage branch August 31, 2026 06:56
@viirya

viirya commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

Thanks @sunchao

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native shuffle has no test coverage for struct data columns

2 participants