rewrite_table_path does not update sizes of rewritten manifests in manifest lists

[vaultah]

Apache Iceberg version

1.9.2 (latest release)

Query engine

None

Please describe the bug 🐞

Iceberg's rewrite_table_path procedure (RewriteTablePath Spark action) does not update sizes of rewritten manifests in manifest lists that reference them.

Specifically, when RewriteTablePathUtil.rewriteManifestList rewrites the manifest list, it copies the contained manifests and only updates their paths. Manifest lists are rewritten first and manifest files are rewritten second, so the utility has no way of knowing final sizes of the rewritten manifest files. This is very similar to #12554.

Mismatch between manifest size in manifest list and size of the physical manifest file may cause correctness problems in some query engines. For example if the actual manifest file is larger than its size recorded in manifest lists, Trino may only read part of it and ignore the rest, effectively skipping manifest entries (referenced data and delete files) silently.

This is reproducible in Iceberg 1.8.1 at least, and based on the source code this is also true for later versions.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[kolban-google]

Impact statement: This issue appears to be breaking Google's BigQuery Iceberg processing of tables that were re-written using rewrite_table_path.

[dramaticlly]

We had a constructive discussion about this issue on iceberg community sync Youtube recording and here's the takeaway

1. With access to target table or ability to read last copied metadata.json for target table is prerequisite to provide incremental path rewrite. Currently we have a gap as existing rewrite table path Spark action solely rely on information available on source table.
2. When target table is not available, rewrite historical manifests are needed in order to have correct manifest size in target table after path rewrite, some but not all engines require correct manifest size in order to perform read on the iceberg table. User opt-in might be needed instead of making this a default operation mode. Previous, we only rewrite manifests are added by the changing snapshots, not we rewrite manifests referenced by the changing snapshots.
3. Other alternatives like repair-manifest API: Define RepairManifests action interface #10784 is also discussed, as it might be easier to directly repair the manifests of target table.

[0xRobin]

We are currently running into this issue, trying to move iceberg table data into GCS and BigQuery throws EOF errors when trying to read it. Is there any known workaround?

[adrians]

Hi @0xRobin,

The workaround that I've found is that after a rewrite_table_path is run (and a table migrated to a new storage path, then registered into the table-catalog), we open a spark-shell using the new catalog and call the "rewrite_manifests" procedure (https://iceberg.apache.org/docs/1.6.0/spark-procedures/#rewrite_manifests). This procedure refreshes the manifest-files, and fills them in using the correct file-lengths.

[kolban-google]

@adrians ... I also tried to use the rewrite_manifests but it didn't seem to help (me). I did notice that the output was:

+-------------------------+---------------------+
|rewritten_manifests_count|added_manifests_count|
+-------------------------+---------------------+
|0                        |0                    |
+-------------------------+---------------------+

... so it may be that the rewrite_manifests in my instance of data didn't see/find anything to improve and hence the rewrite was nothing and the invalid file sizes remain present.