Core: V4 write direction wrappers by stevenzwu · Pull Request #16936 · apache/iceberg

stevenzwu · 2026-06-23T06:05:07Z

What

Adds v4 write-direction TrackedFile adapters: reusable forwarding wrappers that present a legacy v2/v3 ContentFile (DataFile/DeleteFile) as a v4 manifest-entry row on the write path, without materializing a fresh struct per row. A single adapter is allocated per writer and re-pointed at each file; content stats are served by a map-backed view (MapBackedContentStats) over the file's stat maps, with copy() producing stable snapshots.

Benchmark summary

JMH microbenchmarks compared two strategies for presenting a legacy file as a v4 row: convert (materialize a fresh struct per row) vs wrap (a reusable wrapper allocated once per writer and re-pointed per row, with zero per-row allocation). Throughput in ops/ms (higher is better); allocation via the GC profiler in B/op (lower is better).

Per-column content stats: wrap runs ~1.6–1.9x faster than convert across all column counts and allocates far less (39,280 vs 98,984 B/op at 200 columns). Convert pays per-column allocation and eager bound decode that the map-backed view avoids.
Fixed envelope (null stats): convert and wrap are within measurement error (9130 vs 8680 ops/ms); wrap still allocates less (256 vs 688 B/op). The envelope's small, fixed field count means the wrapper's positional dispatch neither meaningfully helps nor hurts.
Combined (envelope + stats): wrap-both is fastest or statistically tied at every column count with the lowest per-row allocation. The column-stats cost dominates and scales with table width, so wrapping stats is what matters; the fixed envelope edge is swamped once real column stats are present.

Net: the reusable wrap model is adopted because it wins or ties everywhere and minimizes per-row allocation — the cost that grows with table width.

See the benchmark doc for full methodology and per-column numbers.

stevenzwu · 2026-06-23T06:14:04Z

+ * <p>REPLACED files are the prior-state entries of v4 REPLACED/MODIFIED pairs and are not live.
+ * Returns null for manifest files written by pre-v4 writers.
+ */
+ default Integer replacedFilesCount() {


Generally, public-interface changes should be minimized and delayed. Justification for keeping these two on the interface in this PR:
In-tree consumers (this PR + close follow-ups):
TrackedFileAdapters.WrappedManifestInfo (this PR) forwards manifest.replacedFilesCount() / replacedRowsCount() when a v4 root manifest reference row is written.
MergingSnapshotProducer.validateAddedDVs (follow-up phase) filters concurrent data manifests by replacedFilesCount > 0 to detect v4 colocated-DV write conflicts — without it, every concurrent data manifest must be deep-scanned.
Convention. The interface-default pattern matches v3's existing extensions for spec-defined optional fields on ManifestFile: firstRowId (row lineage), keyMetadata (encryption), containsNaN (partition NaN). v4 spec-defined manifest_info counts belong on the same interface in the same shape.
A TrackedFile-based parallel v4 API was considered. Would require rewriting ManifestGroup and every engine that consumes FileScanTask. The cost is disproportionate to the savings.

Projected new default methods across the full v4 stack, all spec-defined additions backing v4 manifest_info / content_entry fields:
In this PR:
default Integer replacedFilesCount()
default Long replacedRowsCount()
default int formatVersion()
when root-level manifest DV bitmaps are implemented:
default ByteBuffer deletedPositions()
default ByteBuffer replacedPositions()

gaborkaszab

Hey @stevenzwu ,
I went through mostly for my own understanding. Left some comments, mostly I'm a bit hesitant to expose setting status and sequence numbers directly through the builders. Let me know what you think!

gaborkaszab · 2026-06-23T14:52:15Z

+
+ private ContentEntryAdapters() {}
+
+ static TrackedFile fromDataFile(


I don't see how the user will use these functions, but would it make sense to merge the fromDataFile and fromDeleteFile into a common fromManifestEntry? Then the user doesn't have to decide if it's data or delete, the adapter can do it instead.

I actually think we should keep them separate. The validation for delete files is different from data files and so is the error handling and messages, I think it's cleaner that way.
The callers also always know which type they have so it should be easier.

I agree with keeping these separate for data and delete file cases.
What I think we need to separate is ManifestEntry and DataFile. We accept DataFile instances through the API and create entires in operations like AppendFiles. The best use case is when accepting data files through the public API. There's a secondary use case for wrapping/adapting ManifestEntry for internal uses like rewriting v3 files to v4, but these are more narrow.
For now, I think we need the DataFile to TrackedFile adapter, not ManifestEntry.

stevenzwu · 2026-07-26T03:04:32Z

 null);
 }
-
- private static DeletionVector deletionVector() {


this moved to a static variable near top.

Add reusable write-direction wrappers that present a legacy DataFile, DeleteFile, or ManifestFile as a v4 TrackedFile row for manifest serialization, re-pointed per row to avoid per-row allocation. Content stats are served by a reusable map-backed view over the file's stat maps, with copy() producing a stable snapshot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- ManifestFile: simplify LEGACY_FORMAT_VERSION and recordCount javadocs; clarify that recordCount is entries, not file records - GenericManifestFile: drop field-level comments (interface javadoc covers them); make the pre-v4 and v4+ constructors self-contained per the codebase convention (BaseSnapshot / GenericPartitionFieldSummary) instead of the pre-v4 delegating to the v4+ variant; drop stale arg-order-conflict note - TrackedFileAdapters: trim class javadoc to the one-line intent; drop redundant field-count and eq-delete validation comments; document the in-place re-point + fluent-return semantics on wrap(...); note that forDataFile / forEqualityDeleteFile accept pre-v4 source files (the formatVersion param gates the target manifest format only); invert sortOrderId so the base returns null and DataTrackedFile overrides; add TODO for wiring REPLACED aggregates on WrappedManifestInfo Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Move MapBackedContentStats out of TrackedFileAdapters into its own file, mirroring the top-level ContentStatsBackedMap on the read side (and its already top-level test). MapBackedFieldStats stays a private nested helper since it reads the parent's stat maps directly. Pure relocation with no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- Drop Serializable from MapBackedContentStats/MapBackedFieldStats. These reusable write-direction wrappers are never Java-serialized (the writer consumes them via StructLike, and their holders ContentTrackedFile / ManifestTrackedFile are not Serializable), so they don't need it and don't require a round-trip serialization test. - Carry recordCount and formatVersion through GenericManifestFile.CopyBuilder for non-GenericManifestFile inputs. The else branch built via the pre-v4 constructor and silently dropped both new v4 fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…eldStats apache/main (via apache#17322) added presence methods to FieldStats. Implement hasValueCount / hasNullValueCount / hasNanValueCount in MapBackedFieldStats, mirroring FieldStatsStruct. Also drop the redundant final class modifiers and make MapBackedFieldStats a non-static inner class, so it reads the enclosing stats view's maps directly instead of carrying an explicit parent reference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Production: - Implement the new FieldStats presence methods (hasValueCount / hasNullValueCount / hasNanValueCount) in MapBackedFieldStats. - Drop the -1 sentinel from the count getters; valueCount / nullValueCount / nanValueCount unbox directly and callers must check has*Count() first (matches FieldStatsStruct after apache#17322). - Drop redundant final modifiers and make MapBackedFieldStats a non-static inner class, reading the enclosing view's maps directly. Tests: - Assert upperBound type in testBoundDecodingPerType. - Add testSetNotSupported for the outer set(); split testFieldStatsCopyAndSetNotSupported into two focused tests. - Sharpen testContentStructLikeGetReturnsChildrenOrNull to verify the null slot maps to source field 5, not just that some position is null. - Split the StructLike surface out of testCountsOnlyColumnOmitsBounds into two symmetric tests (testDefaultStructLike / testCountsOnlyStructLike) using Comparators.forType with TestHelpers.Row expectations. - Extend testCountsAndDefaults to also cover the absent-count throw path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@PARAM

TrackedFileAdapters: - Move final manifestInfo field to the top of ManifestTrackedFile so all final fields precede non-final ones. - Trim redundant Javadoc: drop "Instantiate once per writer / call wrap on each row" and "returns this for fluent usage" phrasing from the factory and wrap() methods, and tighten the @PARAM docs (formatVersion "must be 4+"; drop "table schema for building X from the file's stats" style restatements while keeping the caller guidance on partitionType). TestMapBackedContentStats: - Add message checks on the auto-unbox NullPointerException assertions so checkstyle's AssertThatThrownByWithMessageCheck rule accepts them. TestTrackedFileAdapters: - Consolidate STATS_SCHEMA and TABLE_SCHEMA into one TABLE_SCHEMA (optional int id + optional float score) that backs both the read-side stat fixtures and the write-side wrapper factories. - Replace the local EMPTY_PARTITION_DATA constant with the shared PartitionData.EMPTY singleton. - Drop the V4_AND_ABOVE parameterization from testDataFileWrapperAdded; every other test in this file uses FORMAT_VERSION_V4 directly. - Move the assertNullTrackingFields / specsById / partition / dummyTrackedFile helpers to the end of the class so all private-static helpers live together after the tests. - Fill assertWriteDataFields coverage (sortOrderId, keyMetadata, manifestInfo/deletionVector/equalityIds nulls). - Fill testDataFileDoubleWrapRoundTrip DataFile-API coverage (content, partition via Comparators.forType, sortOrderId, splitOffsets, keyMetadata, firstRowId, nanValueCounts); document the two round-trip losses (columnSizes -> null, empty nan map -> null). - Reorder v4WriteManifestFile's last two params to match the trailing positions of the v4 GenericManifestFile constructor; add the missing status assertion in testManifestReferenceWrapperForV4Manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@test

Added earlier in this PR for a parameterized test that was later converted to a plain @test; the constant now has no callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…avadoc The "reusable wrapper" phrasing in the summary sentence already conveys the one-per-writer usage; the trailing sentence just restates it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The class summary "Reusable view over a legacy ContentFile's stat maps" already conveys the reuse contract; drop the follow-up sentence that restated it and described the wrap() mechanism / perf rationale. Keep the lazy-decode note and the copy-unsupported guidance since those are load- bearing for callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The lazy-decode note and the copy-unsupported guidance describe internals of a package-private class. The copy() method itself throws with the "materialize via a writer instead" guidance, so callers who try it get the same message at runtime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

site/docs/contribute.md asks newly added test methods to omit the `test` prefix. The class already omits `public` on the class and its methods, so this brings the remaining half of that convention in line. testType becomes typeMatchesStatsReadSchema rather than a bare type(): the mechanical strip would leave a name that describes no behavior and reads confusingly next to the stats.type() call it asserts on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rebasing PR 16936 onto main picked up two incompatible upstream changes: - TrackedFile.schemaWithContentStats(StructType, StructType) was renamed to TrackedFile.schema(...) and now returns Schema instead of Types.StructType, so TRACKED_FILE_FIELD_COUNT goes through asStruct(). - The TrackedFileStruct constructor moved the partition parameter from position 6 to position 9, after recordCount/fileSizeInBytes/specId. The dummyTrackedFile test helper is moved to the end of the file by this PR, which kept the pre-reorder argument order past the rebase. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

anoopj · 2026-08-04T00:19:43Z

+
+ // Presents a TrackedFile as its persisted StructLike, shared by the reusable write-direction
+ // wrappers.
+ private static Object getByPos(TrackedFile file, int pos) {


The getByPos() here mirrors TrackedFileStruct.getByPos(). Since both track TrackedFile.schema() field order, is there a risk they drift when a field is added?
I don't have great solutions to avoid this duplication though. We probably don't want TrackedFile.getByPos to call into this method with a self pointer since we want the structs to be self contained without any dependencies on adapter.

Worth fixing. One clarification on the failure mode: adding a field is the loud case — size() derives from TrackedFile.schema() on both sides, so a new ordinal falls through to default -> throw. Reordering is the silent one: sizes still match, every ordinal resolves, and values land in the wrong columns.
The mapping shouldn't live on TrackedFile — positional access is a StructLike concern the interface doesn't model. TrackedFileStruct is the better home, since it owns BASE_TYPE, which defines the ordinal domain. Plan is a package-private static Object getByPos(TrackedFile file, int pos) there, called from both. The dependency then runs adapter -> struct, so the structs keep no dependency on the adapter.
Close to mechanical, since the copy here is already static getByPos(TrackedFile, int) invoked as getByPos(this, pos), and the struct's accessors for ordinals 0-12 are plain field returns (partition() returns partitionData, exactly what case 8 reads), with 13-15 already delegating to accessors.
Consolidating also settles the coverage question without new tests: TestTrackedFileStruct.getByPosition already resolves all 16 positions by field name from the schema, so the single remaining switch stays pinned to the schema order. Worth noting internalSet and the hand-written BASE_TYPE still restate that order separately, so this removes one duplicate encoding rather than all of them — though both then sit next to the consolidated getter.

TrackedFileAdapters and TrackedFileStruct each carried a switch mapping positions to TrackedFile fields, so the two could drift as the schema changes. Adding a field is the loud case, since size() derives from TrackedFile.schema() on both sides and a new ordinal falls through to the default branch. Reordering is the silent one: sizes still match and every ordinal resolves, so values would land in the wrong columns. Keep the mapping in TrackedFileStruct, which owns BASE_TYPE and therefore defines the ordinal domain, as a package-private static taking a TrackedFile. Positional access is a StructLike concern that the TrackedFile interface does not model, so the interface is not the right home. The dependency now runs adapter -> struct, leaving the structs free of any adapter dependency. TestTrackedFileStruct.getByPosition already resolves all 16 positions by field name from the schema, so the single remaining switch stays pinned to schema order without new tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CTTY · 2026-08-25T22:48:21Z

+ @Override
+ public TrackedFile copy() {
+ throw new UnsupportedOperationException(
+ "Reusable content-file wrapper does not support copy(); materialize via a writer instead");


Trying to understand better here: does this mean the writer needs to materialize the adapter before writing the current tracked file, so the adapter can be reused by the next writer? This means the writer will need to use the adapter synchronously, correct?

rdblue · 2026-09-03T23:51:46Z

 int PARTITION_SUMMARIES_ELEMENT_ID = 508;

+ /** Format version for pre-v4 manifest files. */
+ int LEGACY_FORMAT_VERSION = 0;


Does this need to be public? Another location would probably be better to avoid polluting the public API.

rdblue · 2026-09-03T23:53:15Z

+ * manifests.
+ */
+ default Integer replacedFilesCount() {
+ return null;


Isn't the pre-v4 value 0 because that status did not exist? That seems like the best thing to return to me. Should this also be required since it is always known for v4 manifests?

rdblue · 2026-09-03T23:56:46Z

 }

+ /** Returns the number of entries in the manifest file, or null for pre-v4 manifests. */
+ default Long recordCount() {


Why is this needed? We should have all of the more specific counts.

rdblue · 2026-09-03T23:57:45Z

+ }
+
+ /** v4+ constructor variant that accepts recordCount and formatVersion. */
+ GenericManifestFile(


I don't think that this should be used for v4. Manifests are now stored as TrackedFile. Why would we update the v3 class used to read from manifest lists?

rdblue · 2026-09-03T23:58:30Z

 private byte[] keyMetadata = null;
 private Long firstRowId = null;
+ private Long recordCount = null;
+ private int formatVersion = LEGACY_FORMAT_VERSION;


Why update this class when these values are correctly provided by the interface?

rdblue · 2026-09-04T00:00:22Z

 static final int MIN_FORMAT_VERSION_ROW_LINEAGE = 3;
 static final int MIN_FORMAT_VERSION_PARQUET_MANIFESTS = 4;
 static final int MIN_FORMAT_VERSION_OPTIONAL_LOCATION = 4;
+ static final int MIN_FORMAT_VERSION_ADAPTIVE_MANIFEST_TREE = 4;


Can we combine this with the min version for Parquet manifests? I don't think that we need multiple constants for different things added in v4.

rdblue · 2026-09-04T00:03:04Z

+ * @param partitionType target partition struct type; use one spec's partition type for a
+ * single-spec manifest, or the union across live specs for a multi-spec manifest
+ */
+ static DataTrackedFile forDataFile(


It looks like this has a different API than the adapters from TrackedFile to ManifestFile and DataFile. Those create a wrapper on each call, while this creates a wrapper that can be reused. I like the reuse, but we should be consistent.
Another issue is that we need to check whether a file being wrapped is already wrapped. For example, if passed a TrackedDataFile, I think this class should simply unwrap it and return the original.

rdblue · 2026-09-04T00:06:06Z

+ /**
+ * Returns a reusable wrapper that presents a {@link DataFile} as a {@link TrackedFile} row.
+ *
+ * @param formatVersion the target table's format version (must be 4+)


I don't think this should be passed. We can always add it later when needed. It also conflicts with how we typically handle writes, which is to have a version-specific wrapper and keep objects in memory using the latest version's expectations.

rdblue · 2026-09-04T00:07:47Z

+ *
+ * @param formatVersion the target table's format version (must be 4+)
+ * @param tableSchema table schema used to build {@link ContentStats} from the file's stats
+ * @param metricsConfig metrics config used to prune the content stats schema


If a schema is needed, I think that this should use the current table manifest schema instead of passing in parts. Do we actually need a schema though? It seems like we want to adapt to any content stats schema in the write wrapper, rather than passing one here.

rdblue · 2026-09-04T00:08:19Z

+ * @param tableSchema table schema used to build {@link ContentStats} from the file's stats
+ * @param metricsConfig metrics config used to prune the content stats schema
+ * @param partitionType target partition struct type; use one spec's partition type for a
+ * single-spec manifest, or the union across live specs for a multi-spec manifest


Similar to my comment above, I would prefer to handle schema in the wrappers.

github-actionsBot added API core labels Jun 23, 2026

stevenzwu commented Jun 23, 2026
View reviewed changes

stevenzwu added this to V4: metadata tree Jun 23, 2026

gaborkaszab reviewed Jun 23, 2026
View reviewed changes

stevenzwu force-pushed the v4_write_direction_wrappers branch 5 times, most recently from c78d6b1 to 828e0c7Compare June 25, 2026 21:35

github-actionsBot added docs build and removed API labels Jun 25, 2026

stevenzwu force-pushed the v4_write_direction_wrappers branch from 828e0c7 to fa86dc1Compare June 25, 2026 22:18

github-actionsBot added the API label Jun 25, 2026

stevenzwu force-pushed the v4_write_direction_wrappers branch 5 times, most recently from fa46d3d to 55f5573Compare June 26, 2026 05:57

stevenzwu mentioned this pull request Jun 26, 2026
Core: Adjustments for TrackedFileBuilder #16964
Closed

stevenzwu commented Jun 26, 2026
View reviewed changes

Comment threadcore/src/main/java/org/apache/iceberg/TrackedFileBuilder.java Outdated

stevenzwu force-pushed the v4_write_direction_wrappers branch 8 times, most recently from b5c92a0 to d5c696bCompare June 28, 2026 06:09

stevenzwu commented Jul 26, 2026
View reviewed changes

stevenzwuand others added 13 commits July 30, 2026 17:58

API: remove unused V4_AND_ABOVE test helper
87ad5c0
Added earlier in this PR for a parameterized test that was later converted to a plain @test; the constant now has no callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Core: drop redundant "Instantiate once per writer" from forDataFile J…
2890eb0
…avadoc The "reusable wrapper" phrasing in the summary sentence already conveys the one-per-writer usage; the trailing sentence just restates it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

stevenzwu force-pushed the v4_write_direction_wrappers branch from 1922cb0 to 4852d17Compare July 31, 2026 03:00

huan233usc mentioned this pull request Jul 31, 2026
API, Core, Parquet: Carry avg value sizes for v4 content stats #17451
Merged

anoopj approved these changes Aug 4, 2026
View reviewed changes

anoopj mentioned this pull request Aug 19, 2026
Core: Add V4 scan task planner #17541
Open

CTTY reviewed Aug 25, 2026
View reviewed changes

rdblue reviewed Sep 3, 2026
View reviewed changes

rdblue reviewed Sep 4, 2026
View reviewed changes


		private ContentEntryAdapters() {}

		static TrackedFile fromDataFile(

Uh oh!

Conversation

stevenzwu commented Jun 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Benchmark summary

Uh oh!

stevenzwuJun 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwuJun 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblueSep 3, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblueSep 4, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

stevenzwu commented Jun 23, 2026•
edited
Loading

stevenzwuJun 23, 2026•
edited
Loading

stevenzwuJun 23, 2026•
edited
Loading

rdblueSep 3, 2026•
edited
Loading

rdblueSep 4, 2026•
edited
Loading