Uh oh!
There was an error while loading. Please reload this page.
[spark] Do not display an unknown partition statistic as a number - #9200
Merged
JingsongLi merged 1 commit intoAug 13, 2026
Merged
Conversation
sundapengforce-pushed
the
fix/spark-format-table-unknown-statistics
branch
from
August 13, 2026 09:59
743db90 to
3cda86dCompareNegative partition statistics mean unreported values according to PartitionStatistics#isKnown. Spark's DESCRIBE PARTITION and SHOW TABLE EXTENDED PARTITION paths displayed those sentinels directly, while SHOW also defaulted absent row counts and byte sizes to zero. Omit unknown values from DESCRIBE partition parameters, render an unknown creation time as UNKNOWN, and create its temporary display statistics only when the byte size is known. SHOW renders missing or negative row counts and byte sizes as UNKNOWN and omits negative statistic parameters. Cover negative, missing, fully reported, and partially reported metadata across the two display paths.
sundapengforce-pushed
the
fix/spark-format-table-unknown-statistics
branch
from
August 13, 2026 10:45
3cda86d to
1780711CompareJingsongLi
commented
Aug 13, 2026
Contributor
+1 |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A negative partition statistic means that the field was not reported;
PartitionStatistics.isKnownis the contract introduced by #9121. Spark's partition-detail commands still displayed negative sentinels directly, andSHOW TABLE EXTENDED ... PARTITIONtreated an absent row count or byte size as an exact zero.This change keeps unknown values out of numeric display output. It is display-only: it does not write statistics back to a catalog or change query planning.
Changes
DESCRIBE ... PARTITIONomits negative statistics from partition parameters, renders an unknown creation time asUNKNOWN, and creates the temporary displayCatalogStatisticsonly when byte size is known. Row count is handled independently.SHOW TABLE EXTENDED ... PARTITIONrenders missing, nonnumeric, or negative row counts and byte sizes asUNKNOWN. Recognized negative Paimon statistic parameters are omitted; unrelated and nonnumeric metadata is preserved.PaimonFormatTableempty-metadata contract.Testing
apache/master(87eb04451d):UnreportedPartitionStatisticsTesthad 2 passing and 4 failing cases, including0 rows, 0 bytesfor absent keys and-1in mixed-known output.UnreportedPartitionStatisticsTest6/6.DescribeTableTest4/4.paimon-spark-commonandpaimon-spark-ut.178071116b.Notes
The regression suite uses a native Paimon table plus catalog/table wrappers to inject negative, absent, and partially reported metadata. The stock in-repo REST catalog currently materializes zeros, so it cannot naturally produce these contract inputs. The suite does not claim to exercise a real
PaimonFormatTableroute.