Skip to content

[core] Avoid global index rebuilds for unrelated column compaction - #9069

Merged
JingsongLi merged 1 commit into
apache:masterfrom
leaves12138:codex/wip-column-sequence-metadata
Aug 20, 2026
Merged

[core] Avoid global index rebuilds for unrelated column compaction#9069
JingsongLi merged 1 commit into
apache:masterfrom
leaves12138:codex/wip-column-sequence-metadata

Conversation

@leaves12138

@leaves12138leaves12138 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What

  • Make data-evolution global-index refresh planning compare the indexed columns' maximum sequence numbers instead of the compacted file's overall maximum sequence.
  • Apply the same decision to both materialized manifest entries and the projected binary manifest scan.
  • Fall back conservatively to the file-level maximum for legacy or malformed column sequence metadata.
  • Cache table schemas and physical file-field layouts while planning refreshes.
  • Project the persisted _WRITE_COLS_SEQUENCES field in the binary manifest path.

Why

Data-evolution compaction rewrites a full normal data file. The file-level maxSequenceNumber advances even when only a non-index column changed, so the previous planner rebuilt an otherwise valid global index.

PR #9313 persists per-column sequence watermarks in DataFileMeta. This PR consumes those watermarks so non-index-column compaction does not trigger index work, while indexed-column updates still refresh the exact affected row range.

The decision no longer depends on retained snapshot history, so it remains correct after snapshots expire and across repeated compactions.

Compatibility

  • Legacy files without column sequence metadata remain conservative and refresh based on the file-level maximum sequence.
  • Missing or malformed positional arrays also fall back to the file-level maximum.
  • Indexed fields absent from a partial-write file do not mark the corresponding index as changed.
  • The existing CompactFileIdentifierSet path remains unchanged for recognizing deleted files during projected manifest scans.

Scope

This is the functional follow-up to merged PR #9313.

It does not change DataFileMeta, compaction metadata generation, serializers, split protocols, or CollectedDeletes.

Tests

  • DataEvolutionGlobalIndexRefreshPlannerTest covers compacted full files, writeCols ordering, row-tracking fields, malformed metadata fallback, and schema caching.
  • SortedGlobalIndexScannerTest covers sequential non-index/index-column updates with compaction, exact refresh row ranges, and snapshot expiration.
  • Materialized-entry and projected binary-manifest planning remain in parity.
  • GenericGlobalIndexScannerTest.
  • Core Spotless and Checkstyle.

@leaves12138
leaves12138force-pushed the codex/wip-column-sequence-metadata branch from c454d0c to 32f5174CompareAugust 10, 2026 07:21
@leaves12138
leaves12138force-pushed the codex/wip-column-sequence-metadata branch 2 times, most recently from ecb805c to c033c5eCompareAugust 18, 2026 07:34
@leaves12138leaves12138 changed the title [core][WIP] Avoid global index rebuilds for unrelated column compaction[core] Avoid global index rebuilds for unrelated column compactionAug 19, 2026
@leaves12138
leaves12138 marked this pull request as ready for review August 19, 2026 09:35
@leaves12138
leaves12138force-pushed the codex/wip-column-sequence-metadata branch from 9ad492b to 9208788CompareAugust 19, 2026 09:49
@leaves12138
leaves12138 marked this pull request as draft August 19, 2026 09:50
@leaves12138
leaves12138 marked this pull request as ready for review August 19, 2026 09:51
@leaves12138
leaves12138 marked this pull request as draft August 19, 2026 09:52
@leaves12138
leaves12138 marked this pull request as ready for review August 19, 2026 09:57
@leaves12138
leaves12138 marked this pull request as draft August 19, 2026 10:12
@leaves12138
leaves12138 marked this pull request as ready for review August 19, 2026 13:06
@leaves12138
leaves12138 marked this pull request as draft August 19, 2026 14:20
@leaves12138
leaves12138 marked this pull request as ready for review August 19, 2026 14:23
@leaves12138

Copy link
Copy Markdown
ContributorAuthor

I have one question:
Does "SplitSerializer" need to add its version by one?
SplitSerializer is used for converting data between java and rust, so in this way, it should be version2-. And SplitSerializer is used for serializing state for some split like FallbackSplit, so it need to be compatible with version-1.
Please review code for SplitSerializer, thanks~

@JingsongLi
JingsongLi marked this pull request as draft August 20, 2026 02:51
@JingsongLi

Copy link
Copy Markdown
Contributor

Converted to draft, "Add an optional positional long[] of per-column maximum sequence numbers to DataFileMeta" should be a separate PR.

@leaves12138
leaves12138force-pushed the codex/wip-column-sequence-metadata branch from c71a8bd to 60170e5CompareAugust 20, 2026 10:16
@leaves12138
leaves12138 marked this pull request as ready for review August 20, 2026 10:16
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit cdf5675 into apache:masterAug 20, 2026
12 of 13 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@leaves12138@JingsongLi