Support Appends with TimeTransform Partitions by sungwy · Pull Request #703 · apache/iceberg-python

sungwy · 2024-05-06T16:22:32Z

Support writes with TimeTransforms.

TimeTransforms are can be supported natively within pyarrow as pyarrow compute functions, without requiring conversions back and forth between Arrow and Python data types.

…he#702)

Bumps [mkdocs-section-index](https://github.com/oprypin/mkdocs-section-index) from 0.3.8 to 0.3.9. - [Release notes](https://github.com/oprypin/mkdocs-section-index/releases) - [Commits](oprypin/mkdocs-section-index@v0.3.8...v0.3.9) --- updated-dependencies: - dependency-name: mkdocs-section-index dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [cython](https://github.com/cython/cython) from 3.0.8 to 3.0.10. - [Release notes](https://github.com/cython/cython/releases) - [Changelog](https://github.com/cython/cython/blob/master/CHANGES.rst) - [Commits](cython/cython@3.0.8...3.0.10) --- updated-dependencies: - dependency-name: cython dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

HonahX

Looks Great! Thanks @syun64

HonahX · 2024-05-07T05:28:30Z

+ def pyarrow_transform(self, source: IcebergType) -> "Callable[[pa.Array], pa.Array]": ...
+
+ @property
+ def supports_pyarrow_transform(self) -> bool:


Instead of adding a public method, how about we maintaining a private list of transform that supports pyarrow transform for the check at the beginning of append. For transforms that does not yet support pyarrow transform, we could throw NotImplementedError in pyarrow_transform
WDYT?

I was following the existing convention for can_transform and preserves_order that are specified as class properties. I'm in favor of keeping it consistent with these

Thanks for sharing the context. My initial concern was that unlike can_transform and preserve_oder which behave differently across transforms, in the end the support_pyarrow_transform would return True for all the transforms, making it a little bit redundant. But on second thought, since we will likely not support pyarrow transform for other transforms in 0.7.0 release, this support_pyarrow_transform can be a useful property in the upcoming release.
We could deprecate this property after we support pyarrow for all transforms later. WDYT?

Yeah I'm on the same page! To add to that - I'm not yet sure if we will be able to support the other Transform classes in support_pyarrow_transform. The key idea here is that we want to use native pyarrow compute functions to apply the equivalent transform without converting the values back and forth between Arrow and Python data types. I'm not yet sure if we'll be able to do the same for BucketTransform for instance, with the existing range of pyarrow compute functions.

Bumps [tqdm](https://github.com/tqdm/tqdm) from 4.66.2 to 4.66.3. - [Release notes](https://github.com/tqdm/tqdm/releases) - [Commits](tqdm/tqdm@v4.66.2...v4.66.3) --- updated-dependencies: - dependency-name: tqdm dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.0.1...3.0.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.3...3.1.4) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

benihildebrand · 2024-05-08T20:20:40Z

+ elif isinstance(source, TimestamptzType):
+ epoch = datetime.EPOCH_TIMESTAMPTZ
+ else:
+ raise ValueError(f"Cannot apply month transform for type: {source}")


"Cannot apply month transform for type" should be referring to 'hour', not 'month'.

Thank you for the check - appreciate it!

Bumps [tenacity](https://github.com/jd/tenacity) from 8.2.3 to 8.3.0. - [Release notes](https://github.com/jd/tenacity/releases) - [Commits](jd/tenacity@8.2.3...8.3.0) --- updated-dependencies: - dependency-name: tenacity dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.25.0 to 0.25.1. - [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases) - [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md) - [Commits](mkdocstrings/mkdocstrings@0.25.0...0.25.1) --- updated-dependencies: - dependency-name: mkdocstrings dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.5.0 to 7.5.1. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.5.0...7.5.1) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.29 to 2.0.30. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [flask-cors](https://github.com/corydolphin/flask-cors) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/corydolphin/flask-cors/releases) - [Changelog](https://github.com/corydolphin/flask-cors/blob/main/CHANGELOG.md) - [Commits](corydolphin/flask-cors@4.0.0...4.0.1) --- updated-dependencies: - dependency-name: flask-cors dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.5.2 to 7.5.3. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.5.2...7.5.3) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.7.1 to 2.7.2. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](pydantic/pydantic@v2.7.1...v2.7.2) --- updated-dependencies: - dependency-name: pydantic dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [requests](https://github.com/psf/requests) from 2.32.2 to 2.32.3. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.2...v2.32.3) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.2.24 to 0.2.25. - [Release notes](https://github.com/Eventual-Inc/Daft/releases) - [Commits](Eventual-Inc/Daft@v0.2.24...v0.2.25) --- updated-dependencies: - dependency-name: getdaft dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

First step towards apache#579

Bumps [moto](https://github.com/getmoto/moto) from 5.0.8 to 5.0.9. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@5.0.8...5.0.9) --- updated-dependencies: - dependency-name: moto dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Fokko

This looks good @syun64 Thanks for working on this, could you rebase to resolve the merge conflicts?

* Update fsspec.py to respect s3.signer.uri property * Add S3_SIGNER_URI constant, add docs --------- Co-authored-by: Fokko Driesprong <fokko@tabular.io>

…4/iceberg-python into transform-partition-writes

Fokko · 2024-05-31T19:04:44Z

@syun64 It looks like something is off. I see a lot of unrelated changes 🤔

sungwy · 2024-05-31T19:12:22Z

@Fokko hmmm yeah I've seen this happen some times and I don't quite understand it - when you do a diff between the two branches, it shows the right changes: main...syun64:iceberg-python:transform-partition-writes

sungwy · 2024-05-31T19:13:23Z

I'll close this PR and reopen it and see if that helps

sungwy added 8 commits May 4, 2024 02:20

checkpoint
0cad231

checkpoint2
96e5533

todo: sort with pyarrow_transform vals
ddfa9ac

checkpoint
1a5327a

checkpoint
e067a28

fix
069f3bd

tests
615d5e3

more tests
c0a0f32

sungwy requested a review from Fokko May 6, 2024 17:06

felixscherzand others added 3 commits May 6, 2024 10:32

Remove trailing slash from table location when creating a table (apac…
d872245
…he#702)

HonahX reviewed May 7, 2024
View reviewed changes

dependabotBotand others added 5 commits May 6, 2024 22:39

adopt review feedback
6a39eda

Makeadd_filesto supportsnapshot_propertiesargument (apache#695)
990ce80

HonahX approved these changes May 7, 2024
View reviewed changes

Add support for categorical type (apache#693)
0508667

benihildebrand reviewed May 8, 2024
View reviewed changes

dependabotBotand others added 9 commits May 8, 2024 15:10

comment
d14e137

Build: Bump mkdocs-material from 9.5.20 to 9.5.21 (apache#719)
4de207d

Build: Bump getdaft from 0.2.23 to 0.2.24 (apache#721)
d02d7a1

Test, write subset of schema (apache#704)
aa361d1

dependabotBotand others added 10 commits May 29, 2024 06:54

Removerecord_fieldsfrom theRecordclass (apache#580)
d3ad61c
First step towards apache#579

Unify to double quotes using Ruff (apache#781)
cf3bf8a

Support CreateTableTransaction for SqlCatalog (apache#684)
0339e7f

Support CreateTableTransaction for HiveCatalog (apache#683)
84a2c04

Support viewfs scheme along side with hdfs (apache#777)
8d79664

Fokko approved these changes May 31, 2024
View reviewed changes

c-thieland others added 13 commits May 31, 2024 17:19

Updatefsspec.pyto respects3.signer.uri property(apache#741)
20f6afd
* Update fsspec.py to respect s3.signer.uri property * Add S3_SIGNER_URI constant, add docs --------- Co-authored-by: Fokko Driesprong <fokko@tabular.io>

checkpoint
5dd846d

checkpoint2
6357193

todo: sort with pyarrow_transform vals
c30a57c

checkpoint
541655f

checkpoint
afe83b1

fix
00ca5f0

tests
511e988

more tests
3b784ab

adopt review feedback
3711b1b

comment
f16d778

Merge branch 'transform-partition-writes' ofhttps://github.com/syun6…
80d5064
…4/iceberg-python into transform-partition-writes

rebase
9f0a92b

sungwy closed this May 31, 2024

sungwy mentioned this pull request May 31, 2024
Support Appends with TimeTransform Partitions #784
Merged

sungwy deleted the transform-partition-writes branch May 31, 2024 20:12

Navigation Menu

Uh oh!

Support Appends with TimeTransform Partitions - #703

Support Appends with TimeTransform Partitions#703
sungwy wants to merge 81 commits into
apache:mainfrom
sungwy:transform-partition-writes

sungwy commented May 6, 2024

Uh oh!

HonahX left a comment

Uh oh!

Uh oh!

Uh oh!

HonahXMay 7, 2024•
edited
Loading

Uh oh!

sungwyMay 7, 2024

Uh oh!

HonahXMay 7, 2024•
edited
Loading

Uh oh!

sungwyMay 7, 2024

Uh oh!

benihildebrandMay 8, 2024

Uh oh!

sungwyMay 8, 2024

Uh oh!

Fokko left a comment

Uh oh!

Fokko commented May 31, 2024

Uh oh!

sungwy commented May 31, 2024

Uh oh!

sungwy commented May 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

Uh oh!

Conversation

sungwy commented May 6, 2024

Uh oh!

HonahX left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HonahXMay 7, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sungwyMay 7, 2024

Choose a reason for hiding this comment

Uh oh!

HonahXMay 7, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sungwyMay 7, 2024

Choose a reason for hiding this comment

Uh oh!

benihildebrandMay 8, 2024

Choose a reason for hiding this comment

Uh oh!

sungwyMay 8, 2024

Choose a reason for hiding this comment

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Fokko commented May 31, 2024

Uh oh!

sungwy commented May 31, 2024

Uh oh!

sungwy commented May 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

HonahXMay 7, 2024•
edited
Loading

HonahXMay 7, 2024•
edited
Loading