Uh oh!
There was an error while loading. Please reload this page.
Spark: Fix type mismatch in SPJ with bucket partition key - #15555
Spark: Fix type mismatch in SPJ with bucket partition key#15555zheliu2 wants to merge 2 commits into
Conversation
When using Storage Partition Join (SPJ) with a bucket partition key on a String column, StructInternalRow.getUTF8String could throw a ClassCastException because it accessed the partition value using CharSequence.class, but bucket transform produces Integer values. Changed getUTF8StringInternal to use Object.class when accessing the underlying struct value, avoiding the ClassCastException when the actual partition value type differs from the source column type. Also added test coverage for bucket transforms on String columns in SPJ tests.
| } | ||
| @TestTemplate | ||
| public void testJoinsWithBucketOnStringAndIdentityColumns() throws NoSuchTableException { |
There was a problem hiding this comment.
testJoinsWithIdentityAndBucketOnStringColumn and testJoinsWithBucketOnStringAndIdentityColumns seem to cover the same code path. Do we need both?
Also, v4.1 only has testJoinsWithIdentityAndBucketOnStringColumn, and v3.4/v4.0 don't have any of the new tests. Should the coverage be consistent across versions?
There was a problem hiding this comment.
Good points @huaxingao! Removed the redundant testJoinsWithBucketOnStringAndIdentityColumns from v3.5, and added testJoinsWithBucketingOnStringColumn andtestJoinsWithIdentityAndBucketOnStringColumn to v3.4 and v4.0 so all four Spark versions have consistent coverage.
Address review feedback: - Remove redundant testJoinsWithBucketOnStringAndIdentityColumns from v3.5 (covers same code path as testJoinsWithIdentityAndBucketOnStringColumn) - Add testJoinsWithBucketingOnStringColumn and testJoinsWithIdentityAndBucketOnStringColumn to v3.4 and v4.0 for consistent coverage across all Spark versions
This PR appears to be generated and submitted by AI without human review. Closing for now. If you'd like to resubmit, please follow our AI contribution guidelines: https://iceberg.apache.org/contribute/#guidelines-for-ai-assisted-contributions |
Summary
Fix
ClassCastExceptionwhen using Storage Partition Join (SPJ) with a bucket partition key on a String column.Problem
When a table is partitioned by
bucket(N, string_column), the bucket transform produces anIntegerpartition value. However,StructInternalRow.getUTF8StringInternal()callsstruct.get(ordinal, CharSequence.class), which assumes the value is always aCharSequence. During SPJ, Spark reads partition values throughStructInternalRow, and this type mismatch causes:This affects all Spark versions (3.4, 3.5, 4.0, 4.1) since SPJ was introduced in Spark 3.4.
Fix
Changed
getUTF8StringInternal()to usestruct.get(ordinal, Object.class)instead ofstruct.get(ordinal, CharSequence.class), then callvalue.toString(). This handles both the normal case (CharSequence from source columns) and the transform case (Integer from bucket transform).Testing
Added
testJoinsWithBucketingOnStringColumnandtestJoinsWithIdentityAndBucketOnStringColumnto all 4 Spark versions for consistent coverage.Update: Per review feedback from @huaxingao, removed redundant test and ensured consistent test coverage across all Spark versions.
Fixes#15349