Update the Iceberg spec for row-level deletes by rdblue · Pull Request #1499 · apache/iceberg

rdblue · 2020-09-23T18:19:04Z

This updates the Iceberg spec with requirements for row-level deletes, as implemented in Java.

rdblue · 2020-09-23T18:20:39Z

FYI @electrum, @openinx. This is the update to document row-level deletes in the spec.

shardulm94 · 2020-09-25T05:05:38Z

-This section details how to encode row-level deletes in Iceberg metadata. Row-level deletes are not supported in the current format version 1. This part of the spec is not yet complete and will be completed as format version 2.
+This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are not supported in v1.
+
+Row-level delete files are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection. It is recommended that delete files are written using the table's default file format.


What is the reason behind the recommendation for delete files and table data to have the same file format?

Most organizations prefer one format to another, so we just want to set the expectation that if a table is written with ORC, deletes should also be written with ORC. That way if there is a problem with the other format, it only affects the people that use it.

For example, we don't use ORC so there may be a problem actually loading and using it in our environment. We don't really exercise it in our integration tests. So I wouldn't want applications built with the assumption that they should be able to write deltas as ORC files all the time because it is allowed by the spec.

If I remember correctly, Hive also allows configuring differential files separately from base files (properties like target file size, row-group size, etc). I think that will also apply in our case (we can figure out details while implementing minor and major compactions).

shardulm94 · 2020-09-25T23:57:16Z


-Manifest list files store `manifest_file`, a struct with the following fields:
+* Equality delete files stored with an unpartitioned spec are applied as global deletes. Otherwise, delete files do not apply to files in other partitions.
+* Position delete files must be applied to data files from the same commit, when the data and delete file sequence numbers are equal. This allows deleting rows that were added in the same commit.


What is the use case for allowing "deleting rows that were added in the same commit"?

Streaming writes need this. Imagine a Flink job that is replacing rows in batches as they come in by some key:

(id=5, data="a") ... (id=5, data="b") ... (id=5, data="c")

To encode the first, it would use an equality delete for id=5, but after that the sink can't delete the newly added row with an equality delete against the same batch because equality deletes would remove all copies, even the newly written one. (That's why equality deletes only apply to strictly older files.)

To delete a row that is in the same batch (same sequence number), we have to use a position delete because those can target a specific row in a file. The writer needs to keep track of possible conflicts and encode deletes against the data files it is writing, but at least it doesn't need to buffer for the entire batch, then remove duplicates, and write all of the rows at once.

aokolnychyi · 2020-09-27T20:13:10Z

+| **`2147483645  _pos`**      | `long`        | Ordinal position of a row in the source data file |
+| **`2147483546  file_path`** | `string`      | Path of a file, used in position-based delete files |
+| **`2147483545  pos`**       | `long`        | Ordinal position of a row, used in position-based delete files |
+| **`2147483544  row`**       | `struct<...>` | Deleted row values, used in position-based delete files |


No, equality delete files put all columns at the top level so there is no need for a "row" field. This is only used in position-based delete files.

Oh, this is actually to optionally store the deleted records alongside file name + position.

aokolnychyi · 2020-09-27T20:14:51Z

+* [_Position deletes_](#position-delete-files) mark a row deleted by data file path and the row position in the data file
+* [_Equality deletes_](#equality-delete-files) mark a row deleted by one or more column values, like `id = 5`
+
+Like data files, delete files are tracked by partition. In general, a delete file must be applied to older data files with the same partition; see [Scan Planning](#scan-planning) for details. Column metrics can be used to determine whether a delete file's rows overlap the contents of a data file or a scan range.


We don't store partition values in positional deletes just yet, do we?

We don't need to, actually. It is stored in the manifest entry.

That's correct. The manifest metadata tracks the scope of a delete file, both its partition and its sequence number.

I didn't mention all of the specifics here because I wanted to keep this section readable as a summary to understand the basics. The full requirements should be in the spec section.

electrum · 2020-09-29T15:47:37Z

-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+#### Equality Delete Files
+
+Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.


may optionally contain additional columns of the deleted row

What is the use case for this?

I think Flink guys raised a point about being able to reconstruct a CDC pipeline from Iceberg table. For example, if someone merges a binlog exported from MySQL into an Iceberg tables, there may be a need to reconstruct this binlog from the Iceberg table at a later point in time.

That being said, I think it should be disabled by default as the use case is very limited and we don't want to increase the size of delete files for no good reason.

This is also an easy way to set the stats for the delete file correctly. Stats are used to filter delete files that cannot match a given data file.

aokolnychyi · 2020-09-30T00:28:17Z

 ## Goals

-* **Snapshot isolation** -- Reads will be isolated from concurrent writes and always use a committed snapshot of a table’s data. Writes will support removing and adding files in a single operation and are never partially visible. Readers will not acquire locks.
+* **Serializable isolation** -- Reads will be isolated from concurrent writes and always use a committed snapshot of a table’s data. Writes will support removing and adding files in a single operation and are never partially visible. Readers will not acquire locks.


Good catch.

aokolnychyi · 2020-09-30T00:32:48Z

 ### Manifests

-A manifest is an immutable Avro file that lists a set of data files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a snapshot, which tracks all of the files in a table at some point in time.
+A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.


Not related to this PR: there is no requirement for it to be Avro, right?

There is right now. We don't support other formats for metadata files in v1 or v2 at the moment.

electrum · 2020-09-30T01:00:28Z

I've thought a lot about equality deletes and I think they are the wrong design for what I see as the common case. Deletes often occur declaratively via a SQL statement such as DELETE FROM t WHERE x = 5. We can execute this very efficiently by simply recording the x = 5 rather than finding and recording all matching rows. That's great.

What doesn't make sense is recording the file name. The delete applies to everything visible in the table. I actually can't think of any reason why we'd want to restrict it to a specific file. One way to solve this is to make the file name optional -- treat it as a special column in the table, that is only present in the delete file if needed as a filter.

If we remove the file name, then the equality delete becomes very small -- we would typically expect only one record per delete operation. At that point, having a separate file is overkill. Reading a separate file during planning for a single record is expensive. It would be better to store the equality delete inline, with the other metadata.

Now that equality deletes are small and inline, we no longer have to limit them to simple equality. Basic range filters would cover common use cases, such as "delete all data older than X".

aokolnychyi · 2020-09-30T01:01:07Z

+
+Equality delete files store any subset of a table's columns and use the table's field ids. The _delete columns_ are the columns of the delete file used to match data rows. Delete columns are identified by id in the delete file [metadata column `equality_ids`](#manifests).
+
+A data row is deleted if its values are equal to all delete columns for any row in an equality delete file that applies to the row's data file (see [`Job Planning`](#job-planning)).


How do we plan to handle nulls as valid values and nulls as no value at all?

See it below.

If column id is part of equality_ids and contains null, we remove records where col IS NULL.

That's correct. And a missing column in a file is read as all null values, using the normal projection rules.

rdblue · 2020-10-01T01:00:46Z

@electrum, it sounds like there are two parts to your concerns that I'll address. First, what is the use of restricting the scope of equality deletes? And second, why not use predicates for global deletes instead of equality files?

What doesn't make sense is recording the file name. . . . I actually can't think of any reason why we'd want to restrict it to a specific file.

Equality deletes aren't restricted to a specific file. The position deletes remove a particular row in a data file, but equality deletes are like predicates applied to entire partitions.

There's good reason to be able to scope an equality delete to a partition. If we accumulated all of the equality deletes globally, then every data file could potentially have a huge number of deletes to apply. For example, consider a CDC stream that is being written to a table that changes rows through UPSERT operations by some row ID. If the table is also bucketed by that row ID, then we can easily reduce the number of deletes than need to be applied to the average data file by a couple orders of magnitude by restricting the scope of a delete to just one partition. This allows us to accumulate and efficiently handle more changes as deletes before it is necessary to compact.

In addition, because deletes are scoped by sequence number, we won't always be able to compact equality delete files together. If all of the deletes were at the table level, then there would necessarily be a lot more delete files to open per data file, even in a maintained table. That puts pressure on compacting deletes into data files, even if there are only a small number of deleted rows (could be one per file). Again, scoping by partition helps us avoid this causing problems.

Basic range filters would cover common use cases, such as "delete all data older than X".

I think this suggestion was mainly motivated by the idea that all equality deletes could be globally scoped, but I think it's a good suggestion for global deletes. It would be really nice to be able to encode deletes like this for certain use cases. And global equality deletes are already a special case. Maybe we should make a way to encode expressions.

My main concerns with this are around overhead. If global deletes are equality deletes, then we can do some amount of work to determine which partitions match and re-encode the delete against those partitions. For example, in the id = 5 case, even if the table is partitioned by time instead of id, I can go find the hours where the id was active based on lower/upper bounds in data files. That would work for lots of tables, where the record ids are correlated with time. But wouldn't be able to rewrite to more targeted deletes if any other expression were used.

We could store expressions instead of equality deletes everywhere to be able to do the rewrite, but that quickly turns into something horrible to apply at read time. A nice thing about equality deletes is that there is a small set of possible schemas (subsets of table columns) and delete files can be combined by unioning filter sets (or merging if rows are ordered).

I think for now I'd opt not to add global delete filters, but if you think it is a good idea I'd be happy to discuss it more.

electrum · 2020-10-02T23:22:19Z

@rdblue Thanks for the detailed and thoughtful response. I'm embarrassed that I misread the specification and thought equality deletes were at the file level. At the partition level makes sense given how manifest files work. However, for their usage by a relational DELETE query, storing them in separate files is still very expensive at read time, since we have to do a file read for a single value.

I can think of a couple of ways to address this cost:

Allow storing the equality delete inline in the manifest (we could have a strict or recommended size limit)
Share the same equality delete file across multiple partitions (is this allowed? how does cleanup work?)

We might also allow predicates in the inline version, as the inline version would be limited to a small number of predicates. Since they are at the partition level, they are naturally filtered to partitions that existed at the time of the delete. Maybe think of this as a third type of delete operation, rather than as a special variant of equality delete.

Equality deletes in external files for CDC operations by row ID is clever and makes sense, as it makes those operations far less expensive (no need to read the original files and join to find the matching rows). I had not considered that type of operation.

rdblue · 2020-10-03T02:14:25Z

However, for their usage by a relational DELETE query, storing them in separate files is still very expensive at read time, since we have to do a file read for a single value.

For a relational DELETE query, I would either encode it as a global equality delete, or scan through data files and use position deletes. Position deletes are the best for readers, so we generally want DELETE FROM to use them. Equality deletes are mostly for cases where you can't find where the matching rows are, like streaming.

And just to be clear, equality delete files can have an arbitrary number of deletes, one per row. While in the worst case it could be a file read for a single delete, we would not expect that in normal operation.

I don't think I would add deletes inline in manifests. That would require a lot of complication in manifests and would greatly expand table-level metadata in some use cases. I'd prefer to keep deletes distributed like data files, instead of centralized like table metadata.

The idea to share some equality delete files across partitions has occurred to me, but I think it is reasonable to either use a global delete (that is written once and shared) or a partition-level delete. We can also write a global delete once and have a background process rewrite to partition-level deletes. If you need to write fast, then use a global. For more targeted reads, use partition-level.

And better yet, use position deletes to minimize overhead. I really don't see equality deletes being used for SQL cases, because position deletes are so much better for readers. If there isn't time to find the data, use a global delete, but then rewrite to position deletes -- not equality -- because that's better as long as you're doing work in the background.

yyanyy

I'm new to the project, and sorry for the naive question/comment in advance.

yyanyy · 2020-10-13T23:03:59Z

+ 3     | NULL        | Grizzly
+```
+
+The delete `id = 4 AND category IS NULL` could be written as the following equality delete file:


Just for my understanding, the example given is just one way of representing it, alternatively it could be

equality_ids=[1, 2] 1: id | 2: category -------|------------- 4 | NULL

is that correct?

Also it sounds to me that for reconstructing a CDC pipeline efficiently, either the delete file or the table may need an additional flag to signal that a delete file (or all delete files) contain all columns of rows, but I guess that's outside of the scope.

Yes, there is no need to include columns other than the ones listed in equality_ids.

for reconstructing a CDC pipeline efficiently, either the delete file or the table may need an additional flag to signal that a delete file (or all delete files) contain all columns of rows

Yes, we may need to signal additional requirements for CDC use cases like writing all known columns into delete files.

yyanyy · 2020-10-13T23:04:02Z

+Row-level delete files are tracked by manifests, like data files. A separate set of manifests is used for delete files, but the manifest schemas are identical.

-#### Position-based Delete Files
+Both position and equality deletes allow encoding deleted row values with a delete. This can be used to reconstruct a stream of changes to a table.


Curious, what's the use case of encoding deleted row values within an equality delete file? I thought one reason of using equality delete file is to not have to read the data file, otherwise we may directly create a position delete file. Or this will only happen when the delete query happen to include all row values of the row it wants to delete? If this is the case, how common is this use case?

In some cases, we may have the full row that is deleted. I think some CDC streams produce both the "deleted" row and the updated row for stream processing, and we don't necessarily know where the "deleted" row was stored in the table (other than its partition).

rdblue · 2020-10-28T22:41:12Z

I'm going to merge this since there isn't a lot of discussion happening. This isn't final yet, so there's plenty of time to discuss more and update. Thanks for the reviews to point out what needed to be clarified, everyone!

Update spec for row-level deletes.

5e52581

probot-autolabeler Bot added the docs label Sep 23, 2020

rdblue requested review from aokolnychyi, cwsteinbach, danielcweeks, omalley, rdsr and shardulm94 September 23, 2020 18:19

rdsr reviewed Sep 24, 2020

View reviewed changes

Comment thread site/docs/spec.md Outdated

shardulm94 reviewed Sep 25, 2020

View reviewed changes

rdblue mentioned this pull request Sep 25, 2020

Fix rendering of lists for format version changes #1507

Merged

shardulm94 reviewed Sep 26, 2020

View reviewed changes

aokolnychyi reviewed Sep 27, 2020

View reviewed changes

electrum reviewed Sep 29, 2020

View reviewed changes

edgarRd reviewed Sep 29, 2020

View reviewed changes

Comment thread site/docs/spec.md Outdated

aokolnychyi reviewed Sep 30, 2020

View reviewed changes

Comment thread site/docs/spec.md Outdated

aokolnychyi reviewed Sep 30, 2020

View reviewed changes

yyanyy reviewed Oct 13, 2020

View reviewed changes

Update for review comments.

85f0b16

aokolnychyi approved these changes Oct 28, 2020

View reviewed changes

rdblue merged commit 8ae9104 into apache:master Oct 28, 2020

This was referenced Oct 28, 2020

Spec: Add delete file partition and sequence number scope rules #361

Closed

Spec: Add column equality delete files #360

Closed

yshcz mentioned this pull request Dec 24, 2025

Spec: Add implementation note for determining manifest list format version #14926

Closed

3 tasks


		Equality delete files store any subset of a table's columns and use the table's field ids. The _delete columns_ are the columns of the delete file used to match data rows. Delete columns are identified by id in the delete file [metadata column `equality_ids`](#manifests).

		A data row is deleted if its values are equal to all delete columns for any row in an equality delete file that applies to the row's data file (see [`Job Planning`](#job-planning)).

Uh oh!

Conversation

rdblue commented Sep 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rdblue commented Sep 23, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shardulm94 Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue Sep 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

electrum commented Sep 30, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue commented Oct 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

electrum commented Oct 2, 2020

Uh oh!

rdblue commented Oct 3, 2020

Uh oh!

yyanyy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rdblue commented Sep 23, 2020 •

edited

Loading

shardulm94 Sep 25, 2020 •

edited

Loading

rdblue Sep 28, 2020 •

edited

Loading

rdblue commented Oct 1, 2020 •

edited

Loading