Uh oh!
There was an error while loading. Please reload this page.
[spark] Add Format Table partition operation tests and clearer errors - #9119
Merged
JingsongLi merged 5 commits intoAug 12, 2026
Merged
Conversation
JingsongLi
commented
Aug 10, 2026
Contributor
CI failed |
MemberAuthor
Local Spark 3.5 / Scala 2.12 evidence:
Fix: 22e5269. |
leaves12138
requested changes
Aug 11, 2026
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
Thanks for adding this coverage. I found two issues that should be addressed before merging.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
leaves12138
approved these changes
Aug 11, 2026
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Thanks for addressing the comments.
TRUNCATE TABLE, TRUNCATE TABLE PARTITION and ALTER TABLE ... RENAME TO PARTITION all refuse a format table with catalog-managed partitions, and these tests pin that they refuse without moving anything: a registration pointing at a directory that is gone, or data under a spec nobody registered, would both be worse than the refusal. The refusal TRUNCATE PARTITION produced described the table wrongly. "Only FileStoreTable supports partitions" is reached by a table that has partitions and lists them through this very trait, which sends the reader looking for the wrong problem; it now says which operations do manage those partitions. A null partition value registers under the default partition name, keeps a directory of that name, reads back as null, is discovered by MSCK REPAIR and is dropped by that name. SHOW PARTITIONS prints it as dt=null rather than the default name, which is what Spark's own ShowPartitionsExec does with a null value; a native Paimon table prints the same thing, so the test records where that behaviour comes from instead of working around it here. The remaining probes cover ADD/DROP PARTITION idempotence, prefix DROP, partial specs in SHOW PARTITIONS, case-insensitive partition column names and empty string partition values, all of which already behaved this way.
sundapengforce-pushed
the
upstream/s7-format-table-partition-operations
branch
from
August 12, 2026 07:03
88d9b82 to
febbaddCompareJingsongLi
commented
Aug 12, 2026
Contributor
+1 |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add Spark SQL end-to-end tests for partition operations on a catalog-managed Format Table: ADD / DROP PARTITION, SHOW PARTITIONS, writer-side registration, MSCK discovery, partial specs, and null / empty / default partition values. TRUNCATE PARTITION and RENAME PARTITION stay unsupported, and the tests pin that a rejected command leaves catalog registrations and rows unchanged.
This PR also clarifies two refusal messages:
Only FileStoreTable supports partitionswas wrong for a partitioned Format Table. The message now says the operation is supported only for a native Paimon table and points Format Table users toADD PARTITION,DROP PARTITION, andMSCK REPAIR TABLE.ADD PARTITIONandDROP PARTITIONon a Format Table with filesystem-discovered partitions now explain that a partition is a plain directory under the table location, to be created or removed directly, and how to enable catalog-managed partitions withmetastore.partitioned-tableandMSCK REPAIR TABLE.No public API, REST contract, or storage format change.
Tests
FormatTablePartitionDdlPlanningTest: 22/22 passed, covering the filesystem-discovery refusal messages.