Skip to content

Python: Alter table plumbing and REST support - #6323

Merged
Fokko merged 30 commits into
apache:masterfrom
Fokko:fd-alter-table
Jun 22, 2023
Merged

Python: Alter table plumbing and REST support#6323
Fokko merged 30 commits into
apache:masterfrom
Fokko:fd-alter-table

Conversation

@Fokko

@FokkoFokko commented Nov 30, 2022

Copy link
Copy Markdown
Contributor

The plumbing to do an table.alter().commit() with actions in between. We can reuse this later on to do multi table commits:

fromdatetimeimportdatetimenow=str(datetime.now())
withTransaction(catalog) astransaction:
table.alter().setProperties(updated_at=now)
table.alter().setProperties(updated_at=now)
# Multi table commit done

Comment threadpython/mkdocs/docs/index.md Outdated
@FokkoFokko closed this Feb 28, 2023
@Fokko
Fokko deleted the fd-alter-table branch April 26, 2023 09:49
@Fokko
Fokko restored the fd-alter-table branch April 26, 2023 09:49
@FokkoFokko reopened this Apr 26, 2023
@Fokko
Fokkoforce-pushed the fd-alter-table branch 2 times, most recently from d4b6a5b to d7c795cCompareApril 28, 2023 10:46
@FokkoFokko added this to the PyIceberg 0.5.0 release milestone Apr 28, 2023
@FokkoFokko changed the title Python: Alter table for table propertiesPython: Alter table plumbing and REST supportApr 28, 2023
Comment threadpython/mkdocs/docs/api.md Outdated
Comment threadpython/mkdocs/docs/api.md Outdated
Comment threadpython/mkdocs/docs/api.md Outdated
Comment threadpython/mkdocs/docs/feature-support.md Outdated
Comment threadpython/pyiceberg/catalog/null.py
Comment threadpython/pyiceberg/catalog/rest.py Outdated
Comment threadpython/pyiceberg/catalog/rest.py Outdated
Comment threadpython/pyiceberg/catalog/rest.py Outdated
Comment threadpython/pyiceberg/table/__init__.py Outdated
Comment threadpython/pyiceberg/table/__init__.py Outdated
Comment threadpython/pyiceberg/table/__init__.py
Comment threadpython/mkdocs/docs/api.md Outdated
Comment threadpython/pyiceberg/catalog/rest.py Outdated

return self.load_table(to_identifier)

def _commit(self, *table_requests: CommitTableRequest) -> CommitTableResponse:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to combine these? Why not have separate commit_table and commit_transaction methods? Then we don't need the check that only one request is supported. I also like that you could previously pass requirements and updates to a public method. Are you trying to restrict access to those in the "public" API for some reason?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, given this is called afterwards in Transaction, it feels more nature to call it just commit_table so we an use self._table.catalog.commit_table instead of self._table.catalog._commit

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thinking of it, that makes sense. Also, the return type is different for a table and a transaction. Thanks!

@rdblue

Copy link
Copy Markdown
Contributor

Looks good to me. We can always update the method signature later since this one is internal. I'd merge, but the lock file has conflicts.

@jackye1995jackye1995 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly looks good to me, +1 for merging so we can start adding implementations for the other catalogs.

Comment threadpython/pyiceberg/catalog/rest.py Outdated

return self.load_table(to_identifier)

def _commit(self, *table_requests: CommitTableRequest) -> CommitTableResponse:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, given this is called afterwards in Transaction, it feels more nature to call it just commit_table so we an use self._table.catalog.commit_table instead of self._table.catalog._commit

ALWAYS_TRUE = AlwaysTrue()


class Transaction:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we plan to have another MiltiTableTransaction for transactions across tables?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather expect a List[Transaction]

@Fokko

Copy link
Copy Markdown
ContributorAuthor

Thank @jackye1995 for chiming in here! 🙏🏻

@Fokko
Fokko merged commit 80aee85 into apache:masterJun 22, 2023
@Fokko
Fokko deleted the fd-alter-table branch June 22, 2023 06:56
@FokkoFokko mentioned this pull request Jun 22, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Fokko@rdblue@jackye1995