Spec: V4 Adaptive Metadata Tree Spec Changes for Entry Structures

[amogh-jahagirdar]

This is one PR for V4 Adaptive Metadata Tree spec changes. The focus of this PR is to update the proposed entry structure in https://docs.google.com/document/d/1k4x8utgh41Sn1tr98eynDKCWq035SV_f75rtNHcerVw/edit?tab=t.wd1z5eeup025#heading=h.80fbnuij9rhg

There will be other PRs for updating the scan planning section, adding implementation notes for CDC etc.

[amogh-jahagirdar]

Two general working principals:

1. Trying to make it easy for someone who just wants to implement v4 to click the v4 tabs, and have it be self describing.
2. Try to do 1 without shifting too much of the existing spec contents, to avoid the risk of changing the spec for v1-v3.

[amogh-jahagirdar]

If people like the two tabbed approach then we should get #14656 in first, that'll make this diff easier to review

[stevenzwu]

nit: most of the time, lower case v1, v2 are used in the spec

[stevenzwu]

Do we want to want to mention colocated DVs and column files for data manifests in v4+?

[rdblue]

This seems reasonable to me.

[stevenzwu]

because partition values are stored as column statistics.

This sentence probably needs to be updated based on the last community sync.

[stevenzwu]

POSITION_DELETES handling — the table omits value 1 entirely, but the comment two paragraphs below ("Value 1 (POSITION_DELETES) no longer applies in entries") is the only place that says so. Worth making the table itself unambiguous: either list 1: RESERVED (writers must not produce; readers must reject) or call out in the table cell that 1 is intentionally skipped. Otherwise a reader reaches for the int and wonders if the spec just forgot it.

[stevenzwu]

V4 writers must produce writer_format_version 1.

why not 4 for V4 writers? null for entries written before V4.

Also, the 0: PRE-V4 enum is odd: PRE-V4 manifests use the manifest_entry schema, not content_entry, so they can't have this field. When does value 0 ever appear?

[gaborkaszab]

this should always be 1, right? Shouldn't we remove this and allocate field ID 157 to column_files? cc @amogh-jahagirdar

[stevenzwu]

the deletion_vector schema is only for external data DVs with location field. the inline manifest DV is stored as part of the manifest_info, which is a bit inconsistent to me. Ideally, the DV struct can capture both outline (data DV) and inline (manifest DV) at the top level. E.g. the cardinality field can be shared.

[stevenzwu]

leaf delete manifest files can also contain entries with position deletes written back when the format version is 2. that also means the line 683 above is also inaccurate

[stevenzwu]

it is better to capture these in the description column in the table (instead of a separate bullet list here).

[stevenzwu]

also need a table for v4

[RussellSpitzer]

Do we want to add anything up front about V4 unifying this structure? I just note that you extended this paragraph to say Data Manifests and Delete Manifests, but now we will only have one type of manifest

[anoopj]

That sounds like a good idea to me.

[RussellSpitzer]

I thought technically we aren't allowing a reference to a delete file in the Root Manifest or in any V4 Manifest except for V4 Delete manifests for equality deletes. Shouldn't it always be a coupled entry of DV and DataFile or DV and Manifest?

[RussellSpitzer]

Not sure why we are changing the pluralization here but the change is ok. Just wondering because we immediately switch back to singular in the next paragraph.

[RussellSpitzer]

I'd leave the rational out of this paragraph. I think it's find to just say that they are not bound to a partition spec, I think partition-spec-id needs a better description here ... The spec id used by the writer when generating this data file?

[RussellSpitzer]

While I support this for simplicity, I know @rdblue still wants to have the Avro option. currently the code in my PR lets you write either in the SDK and i'm not sure it is much more expensive to allow both in the spec. Worth having a community discussion though.

[RussellSpitzer]

Why is there write and read? What would optional "read" be? I assume this was probably an LLM just trying to make a balanced table.

More importantly, do we want to relax the write requirements? In V2/3 these were all required, but now they are optional

[RussellSpitzer]

Not sure this one makes sense now? Entries should all have a spec, but i'm not sure it makes sense to have a global spec id for the manifest anymore?

[amogh-jahagirdar]

Miss on my part, yes manifests are no longer bound to a partition spec!

[RussellSpitzer]

In V4 this is data or "equality deletes" but I think it's fine to just call it deletes

[anoopj]

We should probably add the MODIFIED status also? Also, would you please update the AMT proposal doc to include the status? (Also we need to move the content from the Colocated entry tab into the main tab?)

[amogh-jahagirdar]

Yeah I still need to update these when I get a chance, will do

[anoopj]

How do we cover the status transitions: e.g. a file/manifest starts with ADDED, but the next commit it flips to EXISTING (unless it got deleted/replaced). Same for MODIFIED -> EXISTING transition.

But for files that live in leaf manifests, we don't want to rewrite manifests for the sake of doing the status transition. For instance, if we modified DVs on a bunch of files, we might write a new leaf manifest with a bunch of MODIFIED entries (and the manifest status itself would have an ADDED?), but in the next commit, the leaf status might change, but the entries inside the leaf stays MODIFIED.

[amogh-jahagirdar]

How do we cover the status transitions: e.g. a file/manifest starts with ADDED, but the next commit it flips to EXISTING (unless it got deleted/replaced). Same for MODIFIED -> EXISTING transition.

I'm not sure if we explicitly need to call out the ADDED/MODIFIED -> EXISTING transitions? I think the only state transitions we need to call out are the ones where it's a net new change in a given snapshot.

If you look at the current spec (not these changes but the ones for v1-v3), we never really needed to call out the explicit EXISTING transition because anything that was a new change, like ADDED/DELETED had explicit mentions. After defining those, it's very reasonably implied and via the naming of the status EXISTING that any carry over should just be marked as EXISTING.

If we want to have EXISTING be this explicit transition I think we can do it but we'll want to be not overly-specified to the point where we prevent certain write patterns (more on that in the next response).

But for files that live in leaf manifests, we don't want to rewrite manifests for the sake of doing the status transition. For instance, if we modified DVs on a bunch of files, we might write a new leaf manifest with a bunch of MODIFIED entries (and the manifest status itself would have an ADDED?), but in the next commit, the leaf status might change, but the entries inside the leaf stays MODIFIED.

Agree, we definitley don't want to have to rewrite manifests just to update statuses, that would be a crazy bad requirement but I don't know if that means we need to codify explicit transition rules for EXISTING in the spec. I just worry that if we do that too strictly it may be misinterpreted and prevent valid write scenarios.

Every additional statement we make in the spec bears the risk of misinterpretation which is where I'm coming from that we need to be careful.

Concretely:

Status just indicates what the status of that file in the tree is (whether it's a data file or a manifest is inconsequential)

Assume we stick with the status quo of not epxlicitly defining when to state transition to Existing and rather it's implied that anything that's not a new change (not ADDED/DELETED/MODIFIED/REPLACED) is EXISTING:
Both options of the following options are valid:

1/ In the scenario you described, a writer can choose to mark the leaf manifest as EXISTING
or
2/ A writer can choose to actually just write a new manifest with EXISTING entries and mark the old manifest as DELETED.

Could we explicitly enumerate every single state transition or find some wording that generalizes across use cases? Probably, but I think I'd rather just be explicit cases where we know specific statuses are required to be produce (like DV changes/column updates producing REPLACED + MODIFIED, file removals explicitly producing DELETED).

That's a smaller state to enumerate and less chance of misinterpretation preventing legitimate write scenarios. Every other status is just naturally implied and readers/change detection should correctly handle those, unless there's some ambiguous case we're worried about.

Let me know what you think @anoopj , cc @stevenzwu@RussellSpitzer@rdblue

[danielcweeks]

I feel like how we manage the state transitions is rather straightforward: transitions only happen when the metadata is modified. So if a leaf file is not being rewritten, all of the status remain as they were when originally produced.

The important aspect of this is that the status of entries within the manifest file represents the state as of the snapshot where the manifest was written. If you later rewrite a manifest and see that there were entries that had a ADDED, MODIFIED or REPLACED they would then transition to EXISTING within the new manifest and that is now the effective state as of that new snapshot.

This isn't different than how v3 and prior work. Consider an append only table that only produces new files and new metadata files (no metadata rewrite). All entries will have a status of ADDED forever. We don't go back and rewrite all of the entries for every commit to be EXISTING.

We shouldn't have to rewrite older metadata, rather we should interpret in context of the when it was written.

[prakharjain09]

If you later rewrite a manifest and see that there were entries that had a ADDED, MODIFIED or REPLACED they would then transition to EXISTING

@danielcweeks is there a typo here? A REPLACED entry can't go to EXISTING. REPLACED is a tombstone and not a live entry.

[prakharjain09]

Can we have a new ADDED leaf as part of commit X, which has:

5 ADDED DataEntry
5 EXISTING DataEntry
5 DELETED DataEntry

My understanding is the above is allowed.

On next commit (vx+1), this needs to become EXISTING.
But then the 5 DELETE DataEntry needs to be hide somehow in the vx+1 tree.

5 ADDS DataEntry
5 EXISTING DataEntry
5 DELETED DataEntry --------------- this should not be visible as they are not part of v`x+1` CDF

How will this work? Does the writer have to mark this leaf as EXISTING with MDV on last 5 rows??

[prakharjain09]

Somewhere we should explicitly mention that REPLACED applies only for data files. It is not applicable for leaf manifest entries?

[prakharjain09]

Snapshot ID where the deletion vector was added. Inherited when null. Must be null when deletion_vector is null.

Can we explicitly talk about whether the line is for data file or a leaf manifest entry or both?

[rambleraptor]

If would be great if we could have a formal definition for leaf manifests in this document. This is the first place that this concept is introduced and we drop the data / delete distinction in other places later on.

[amogh-jahagirdar]

This field is fine but for sequence number inheritance to work correclty for column updates, I think we'll also need a latest_column_file_sequence_number? cc @anuragmantri@gaborkaszab@anoopj@rdblue

[gaborkaszab]

I believe we agreed to bump the base file's sequence number (the one right below) repurposing it for column files. With this we could also avoid applying the previous eq-deletes for that file (we rewrite them to DVs anyway when adding a column file). Does this make sense, @amogh-jahagirdar ?

I'm not entirely sure what we discussed for file_sequence_number. Should that be untouched when adding a column file opposed to (data) sequence number?

[amogh-jahagirdar]

Hah yeah I realized I replied long ago on just reusing the existing data sequence number and the requirement to just rewrite eq. as DV. https://lists.apache.org/thread/gqcnoh5mrohg129dzl1qwflhfgxytow4 and I had forgotten my reasoning here :).

Ok though, when I was having offline conversations with others I was under the impression that people were thinking that there was another seq. number specific to last updated for columns. But let me confirm with folks (maybe just my misunderstanding) and get this wording tightened up.

[amogh-jahagirdar]

@gaborkaszab Confirmed, what we discussed on the mailing list a few months ago is still the consensus. Was just my misinterpretation of a statement in an offline conversation, we can reuse data sequence number here. I do need to add the rule that it should be set to the snapshot sequence number when there's a column update (and the rule you mentioned about rewriting equality deletes as DV). CC @anoopj .

[gaborkaszab]

Thanks for the confirmation, @amogh-jahagirdar !
Note, I recall we said rewriting any non-dv delete including pos-deletes into a DV.

[amogh-jahagirdar]

Not really sure this needs to be said

[rdblue]

Suggested change

Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file's partition data, and its metrics. The data in a snapshot is the union of all files in its manifests. Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. In v1-v3, data manifests and delete manifests are separate and can track files with any subset of a table; they are not associated with partitions. In v4, a data file and its deletion vector are stored in the same entry, so there is only a single manifest type, described below.

Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file's partition data, an optional deletion vector (v4), and its metrics. The data in a snapshot is the union of all live rows in live data files tracked by the snapshot's manifests. Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing.

• I think we should add the co-located DV to the first sentence, rather than at the end
• We may also want to note that the data in the table is live rows in live data files?
• I would remove the v1-v3 sentence since it is complicated to explain the differences here. This only needs to be high level. We also no longer have the confusion over a manifest-per-partition so we can remove it.

If we still want to note that there are delete manifests, maybe a separate paragraph that covers when they were used would make this easier to read. Something like this:

In v2 and v3, row-level deletes could be added using delete manifests tracking delete files. Existing delete files and delete manifests are valid in v4, but cannot be written and must be replaced by co-located deletion vectors.

[rdblue]

I think this is where I'd introduce a Snapshot root or Root metadata file that we can use to refer to both a manifest list or a root manifest.

[rdblue]

Suggested change

	***Delete manifest** -- (v1-v3 only) A file that lists delete files; a subset of a snapshot.
	***Delete manifest** -- A file that lists delete files to be associated with data files at planning time.

No need to mention v1-v3 here because they are still valid in v4.

[rdblue]

"Manifests store data files from any partition" is true, so I'd like to keep the invariant.

Manifests store data files from any partition. v4 manifests may store partitions from any spec, but manifests in v3 and earlier store files for a single spec.

Also, why remove "a tuple of"? That is specifying that there are multiple values, not a single value. This is distinguishing Iceberg from Hive, where a "partition" was a string: "category=explosives/dt=2021-03-09".

[rdblue]

Similar to the comment above, I'd phrase this first using what is common:

Manifests are tracked by a root metadata file for each table snapshot. In v4, the root metadata file is a root manifest that may track data files in addition to leaf manifest files.

[rdblue]

Suggested change

- v1-v3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.

- v1-v3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec.

I would move the last sentence out to its own paragraph:

The partition spec used when writing each data file is used to transform predicates on the table’s data rows into predicates on partition values during job planning. In v3, the same partition spec is used for all data files in a manifest.