Skip to content

[spark] Do not display an unknown partition statistic as a number - #9200

Merged
JingsongLi merged 1 commit into
apache:masterfrom
sundapeng:fix/spark-format-table-unknown-statistics
Aug 13, 2026
Merged

[spark] Do not display an unknown partition statistic as a number#9200
JingsongLi merged 1 commit into
apache:masterfrom
sundapeng:fix/spark-format-table-unknown-statistics

Conversation

@sundapeng

@sundapengsundapeng commented Aug 13, 2026

Copy link
Copy Markdown
Member

Summary

A negative partition statistic means that the field was not reported; PartitionStatistics.isKnown is the contract introduced by #9121. Spark's partition-detail commands still displayed negative sentinels directly, and SHOW TABLE EXTENDED ... PARTITION treated an absent row count or byte size as an exact zero.

This change keeps unknown values out of numeric display output. It is display-only: it does not write statistics back to a catalog or change query planning.

Changes

  • DESCRIBE ... PARTITION omits negative statistics from partition parameters, renders an unknown creation time as UNKNOWN, and creates the temporary display CatalogStatistics only when byte size is known. Row count is handled independently.
  • SHOW TABLE EXTENDED ... PARTITION renders missing, nonnumeric, or negative row counts and byte sizes as UNKNOWN. Recognized negative Paimon statistic parameters are omitted; unrelated and nonnumeric metadata is preserved.
  • Empty partition-metadata maps keep their existing behavior. This PR does not change metadata producers or the PaimonFormatTable empty-metadata contract.

Testing

  • RED on apache/master (87eb04451d): UnreportedPartitionStatisticsTest had 2 passing and 4 failing cases, including 0 rows, 0 bytes for absent keys and -1 in mixed-known output.
  • GREEN with this change: UnreportedPartitionStatisticsTest 6/6.
  • Spark 3 catalog-managed partition regression suites 29/29.
  • Spark 3.5 DescribeTableTest 4/4.
  • Spark 3 / Scala 2.12 clean reactor compilation and targeted suite 6/6 with JDK 17.
  • Spark 3 / Scala 2.13 clean reactor compilation and targeted suite 6/6 with JDK 17.
  • Spark 4 / Scala 2.13 clean reactor compilation and targeted suite 6/6 with JDK 17.
  • Spotless check for paimon-spark-common and paimon-spark-ut.
  • GitHub Actions: 12/12 checks passed on 178071116b.

Notes

The regression suite uses a native Paimon table plus catalog/table wrappers to inject negative, absent, and partially reported metadata. The stock in-repo REST catalog currently materializes zeros, so it cannot naturally produce these contract inputs. The suite does not claim to exercise a real PaimonFormatTable route.

@sundapeng
sundapengforce-pushed the fix/spark-format-table-unknown-statistics branch from 743db90 to 3cda86dCompareAugust 13, 2026 09:59
Negative partition statistics mean unreported values according to
PartitionStatistics#isKnown. Spark's DESCRIBE PARTITION and SHOW TABLE EXTENDED
PARTITION paths displayed those sentinels directly, while SHOW also defaulted
absent row counts and byte sizes to zero.
Omit unknown values from DESCRIBE partition parameters, render an unknown
creation time as UNKNOWN, and create its temporary display statistics only
when the byte size is known. SHOW renders missing or negative row counts and
byte sizes as UNKNOWN and omits negative statistic parameters.
Cover negative, missing, fully reported, and partially reported metadata across
the two display paths.
@sundapeng
sundapengforce-pushed the fix/spark-format-table-unknown-statistics branch from 3cda86d to 1780711CompareAugust 13, 2026 10:45
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit c9eb26e into apache:masterAug 13, 2026
12 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@sundapeng@JingsongLi