Uh oh!
There was an error while loading. Please reload this page.
[core][spark] Support collecting format table partition statistics in MSCK REPAIR TABLE - #9297
Merged
JingsongLi merged 1 commit intoAug 21, 2026
Conversation
sundapengforce-pushed
the
upstream/s5-measure-partitions-in-msck
branch
10 times, most recently
from
August 19, 2026 12:48
2d03efe to
69432a4Comparesundapengforce-pushed
the
upstream/s5-measure-partitions-in-msck
branch
from
August 19, 2026 17:26
69432a4 to
00290d6CompareJingsongLi
marked this pull request as draft
August 20, 2026 01:05
sundapengforce-pushed
the
upstream/s5-measure-partitions-in-msck
branch
4 times, most recently
from
August 20, 2026 10:39
fe61d73 to
6a78c95Comparesundapeng
marked this pull request as ready for review
August 20, 2026 11:40
sundapengforce-pushed
the
upstream/s5-measure-partitions-in-msck
branch
from
August 20, 2026 11:44
6a78c95 to
3a252e1Compare… MSCK REPAIR TABLE A commit reports what it wrote. Nothing reports what is already there, and for a format table plenty is: partitions written by something that is not Paimon, files deleted out of band, an increment redelivered and counted twice. MSCK REPAIR TABLE is already the command that reconciles the partition set against the directories, so it is the natural place to reconcile the numbers too. Off by default, behind spark.paimon.format-table.repair.collect-statistics, because measuring changes what a repair costs: the plain diff lists partition directories, and measuring lists the files inside every one of them. That is a different order of magnitude on a table with many partitions, and a repair should not silently become that. When it is on, spark.paimon.format-table.statistics.parallelism caps how many partitions are measured at once, at 8: listing one is a round trip the driver spends waiting on, and the cap keeps a table with many partitions from turning that wait into a burst of requests. When on it measures every partition that ends up registered with a directory behind it, not only the ones it just added — the stale numbers of partitions written outside Paimon are exactly what a repair exists to correct. Without ADD it stays inside the already-registered set, so measuring never registers a partition the command was not asked to. The collector reports what a reader would see. File count, byte size and last file creation time come from the listing. It stops there. The row count it leaves unknown: reading it means opening every file's footer, which no listing does and which the command that wants exact rows can pay for on its own. A listing failure aborts the whole collection rather than reporting what it managed to see, because a truncated listing is indistinguishable from a partition that lost files. A partition whose directory is gone measures as an exact zero, with no last file to date. Tests: FormatTablePartitionStatsCollectorTest covers staging trees, hidden files beside the data, a missing directory, the value-only layout and a value the directory name has to escape, alignment with the given specs, a spec missing a partition key, and a listing failure aborting both the serial and the parallel path. FormatTablePartitionRepairTest covers measuring every partition on disk, a repair without ADD registering nothing, and a listing failure leaving the catalog untouched. CatalogManagedPartitionMsckRepairTest covers the command end to end, with the option off and on.
sundapengforce-pushed
the
upstream/s5-measure-partitions-in-msck
branch
from
August 20, 2026 11:49
3a252e1 to
e76a378CompareUh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
A commit reports only what it wrote. A format table also holds partitions written by something that
is not Paimon, files deleted out of band, and increments that drifted, and nothing corrects those.
MSCK REPAIR TABLEalready reconciles the partition set against the directories, so it is where thenumbers can be reconciled too.
Behind
spark.paimon.format-table.repair.collect-statistics, off by default: a plain repair listspartition directories, while measuring lists the files inside every one of them, a different order of
magnitude on a table with many partitions.
spark.paimon.format-table.statistics.parallelism(8) bounds how many partitions are measured at once.
FormatTablePartitionStatsCollectormeasures through the same listing the scan uses, so ameasurement counts exactly the files a reader would return and committer staging trees are pruned
rather than walked. It covers every registered partition that has a directory, not only the newly
added ones, and never registers one the repair did not ask for. A measurement is of a whole
partition, so it replaces what the catalog held rather than adding to it, and a partition whose
directory is gone measures as an exact zero on the file numbers. A listing failure aborts the whole
collection rather than write a partial measurement. The row count is not part of this: reading it
means opening every file's footer, which no listing does.
Tests
FormatTablePartitionStatsCollectorTest: staging trees, hidden files beside the data, a missingdirectory, the value-only layout and a value the directory name has to escape, alignment with the
given specs, a spec missing a partition key, and a listing failure aborting both the serial and
the parallel path.
FormatTablePartitionRepairTest: measuring every partition on disk, a repair without ADDregistering nothing, and a listing failure leaving the catalog untouched.
CatalogManagedPartitionMsckRepairTest: the command end to end, with the option off and on.