[core] Add an optional positional long[] of per-column maximum sequence numbers to DataFileMeta

[leaves12138]

What

• Add an optional positional long[] of per-column maximum sequence numbers to DataFileMeta.
• Reuse writeCols to resolve array positions; full-column files use the file schema field order and row-tracking system fields are excluded.
• Propagate per-column sequence watermarks through data-evolution normal compaction by stable field id.
• Persist the array only for global-index.column-update-action=IGNORE, and store null when every column already equals the file-level maximum.
• Preserve the metadata across manifest, commit-message, split, compaction-task, Flink checkpoint, and Spark copy compatibility paths.
• Return defensive copies from POJO metadata.

Why

Data-evolution compaction rewrites a full normal data file, so the file-level maxSequenceNumber cannot identify which physical columns changed. Snapshot history is not a reliable source for this information because snapshots may expire.

Persisting column-level sequence watermarks in DataFileMeta provides durable metadata for consumers which need to distinguish column updates across repeated compactions.

When all physical columns equal the file-level maximum, null is semantically equivalent because readers fall back to that maximum. Tables not configured with update action IGNORE do not persist the array.

Compatibility

• The manifest field is nullable. Files written before this change fall back to the file-level maximum sequence.
• Missing or malformed positional arrays also fall back to the file-level maximum.
• New Avro manifest files remain readable by the previous 20-field reader schema.
• Add legacy serializers for the previous 20-field DataFileMeta and nested ManifestEntry layouts.
• Bump and route compatibility for CommitMessage, DataSplit, IncrementalSplit, and ChainSplit.
• SplitSerializer remains at v1; its existing v1 golden files are refreshed for the current embedded DataFileMeta layout.
• Flink pending-split checkpoint recovery covers legacy IncrementalSplit v1 and ChainSplit v2 state.
• Spark sys.copy clears the positional array when assigning a different schema id.

Scope

This PR only introduces and propagates the durable DataFileMeta metadata. It does not change global-index refresh planning or scanner decisions. Those changes remain in follow-up PR #9069.

This was split from #9069 following review feedback.

Tests

• Core metadata, manifest, commit-message, split, and compatibility suites: 109 tests.
• Data-evolution compaction E2E coverage for repeated partial-column updates, redundant-array elision, and non-IGNORE behavior.
• Flink task serializer and pending-split checkpoint recovery: 7 tests.
• Spark CopyFilesUtilTest.
• Spotless and Checkstyle for API, core, Flink common, and Spark common.

[JingsongLi]

_WRITE_COLS_SEQUENCES

[leaves12138]

OK

[JingsongLi]

+1

[weijietong]

@JingsongLi@leaves12138 can you explain why we need this version number ? Can we remove this version number and the following version checks? It's a runtime query task's object ser/de-ser . Maybe we could use protobuf to do this work and keep schema evolution compatiable. It's urgly to see this version change.