Can't add column with the same name as a deleted and previously partitioned key column

[madeirak]

Apache Iceberg version

1.4.3

Query engine

Spark

Please describe the bug 🐞

1. column d is also a partition field
   

2. delete column d
   ALTER TABLE test_p DROP partition field d;
   ALTER TABLE test_p DROP COLUMN d;
   

3. add column d;
   

why i got this exception, and what should i do ??

[amogh-jahagirdar]

Thanks for reporting @madeirak . I think this is similar to #10234 but this particular issue also brings up a nuance in the "unreferenced partition spec" case.

One problem imo is we should not allow dropping columns which are part of historical partition specs.
Now in your case it's actually still not referenced since no data was actually written (just the metadata updates) so this is another case to think about how to handle.

Looking at the PartitionSpec/TableMetadata builder code path, what happens is when the schema is updated, all of the historical partition specs and sort orders need to be updated for the new schema. When you attempt to add the column back it'll have a new id and that id won't align with what was on the historical partition spec; this means updating the historical spec which still referenced the old column will fail.

One approach that comes to mind for handling this case is if a historical partition is unreferenced by any manifests we should remove that from the metadata because it won't be needed or useful (unless I'm missing something). The tradeoff is complexity in the metadata building logic, but it seems worth it to handle this case. Though I do wonder what others think?

[amogh-jahagirdar]

Well probably we want to keep the historical spec around, but at least in validation we may want to skip that if it’s unreferenced

[madeirak]

Well probably we want to keep the historical spec around, but at least in validation we may want to skip that if it’s unreferenced

thx for reply, btw, the situation I mentioned above is going to solved by skipping validation or something, right? :)
After this issue is resolved, please @ me

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'