Feature: Write to branches by vinjai · Pull Request #941 · apache/iceberg-python

vinjai · 2024-07-18T17:04:14Z

Fixes: #306

(cherry picked from commit ad1ec67)

(cherry picked from commit 40cf10e)

(cherry picked from commit 6dbe68d)

(cherry picked from commit ad1ec67)

(cherry picked from commit 40cf10e)

(cherry picked from commit 6dbe68d)

vinjai · 2024-10-16T08:15:38Z

@sungwy @kevinjqliu
This PR is ready for review

vinjai · 2024-10-16T20:30:59Z

Fixed another bug.
Writes with same name but different ref types were being successful. This is a bug in the current release version too

Please review whenever you get some time.

kevinjqliu · 2024-10-19T18:33:44Z

Thanks for the contribution! I'll take a look.
I remember adding support for branch is complicated since we need to consider different edge cases.

vinjai · 2024-10-19T19:24:02Z

I have mostly tried to cover all edge cases.
The idea is that the branch is just another iceberg table where the snapshots append independently of the main branch.

I also agree with your concern.
If it helps, we can add more test cases

vinjai · 2024-11-07T22:23:16Z

Hey @kevinjqliu
Did you get a chance to look at this?

kevinjqliu

Thanks for the PR! And sorry for the delay, I was running the 0.8.0 release.
Generally LGTM, I left a comment about add more tests integrating with Spark

vinjai · 2024-11-14T01:09:08Z

Thank you for the review @kevinjqliu

vinjai · 2024-11-20T07:17:05Z

@kevinjqliu What are the next steps to get this merged?

vinjai · 2025-05-23T15:22:06Z

Thanks @SebastienN15 — I’ll review your commit and take it forward from there.
I’ll share the working PR tomorrow.

vinjai · 2025-05-26T04:30:02Z

Identified and fixed a bug related to empty tables.
Planning to add test cases to cover this scenario.

vinjai · 2025-05-27T14:28:55Z

Hey @Fokko
This PR is ready for review again

vinjai · 2025-06-01T16:45:37Z

Hey @Fokko
I noticed that the integration tests for Python 3.9 are failing.
Possibly, a runtime issue:

ServerError: NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: 18436A88DE8BCC82, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8) (SDK Attempt Count: 1)

I ran the same tests locally with Python 3.9.6, and they passed without issues.
Is there any change required in the PR to resolve the above failure?

Fokko · 2025-06-01T19:48:48Z

@vinjai Thanks for pinging me. The issue has been fixed in #2049. Locally, you're still using an older version of minio, and therefore the tests pass. Could you pull in the latest main branch to fix the CI? Thanks!

vinjai · 2025-06-03T09:20:34Z

Hey @Fokko
I've pulled the latest changes from the main branch. Please re-trigger the workflow.

Fokko

Thanks @vinjai for working on this, and sorry for the long wait. I think this looks great, and let's move this forward 👍

Fokko · 2025-06-18T14:51:49Z

+ if branch is None:
+ files = self._scan(row_filter=delete_filter, case_sensitive=case_sensitive).plan_files()
+ else:
+ files = self._scan(row_filter=delete_filter, case_sensitive=case_sensitive).use_ref(branch).plan_files()


Maybe in a subsequent PR we can pass in the ref to the constructor 👍

Fokko · 2025-06-18T15:03:42Z

@vinjai I'm sorry, can you resolve the conflicts once more? I'll merge right after

SebastienN15 · 2025-06-23T14:56:38Z

Hey @vinjai, would you be able to resolve the conflicts? I would love to see this PR merged 🙏

vinjai · 2025-06-24T01:56:38Z

Hey @Fokko
I have resolved all conflicts and merged the recent changes.
While running integration tests, I am facing this error for a few tests:

java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (360448)
Allocator(toArrowBatchIterator) 0/360448/360448/9223372036854775807 (res/actual/peak/limit)

The same error is coming on the main branch too.

Any idea on how to resolve this?

vinjai · 2025-06-28T20:33:30Z

Hey @Fokko,

I tried the tests on a different machine. The tests are running fine.
Request you to run the workflow and merge this PR ahead.

Thanks

Fokko · 2025-07-02T22:50:53Z

@vinjai Thanks, let's move this forward. Thanks @SebastienN15 for pinging me on this matter :)

Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

jean-humann · 2025-07-24T11:40:05Z

Thanks @everyone for this amazing job, @Fokko when could we expect a release with this feature included please ?

kevinjqliu · 2025-07-24T17:00:27Z

@jean-humann we're working on the next release, 0.10, right now! https://lists.apache.org/thread/o2wcs9774rm1krswjgm55sjhgdczyzxl

the release candidate should be out this week or next :)

averyrobbins · 2025-07-25T02:30:51Z

Thank you for adding this feature! Excited to use branches in pyiceberg. Does this PR include the ability to "fast_forward" a branch to the main branch, or has that not yet (hopefully yet) been implemented? Thank you. @Fokko @kevinjqliu

Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

kevinjqliuand others added 4 commits July 15, 2024 03:36

s/"main"/MAIN_BRANCH
d4ca653
(cherry picked from commit ad1ec67)

replace string literals
0b7aaaf
(cherry picked from commit 40cf10e)

default writes to main branch
23f04ec
(cherry picked from commit 6dbe68d)

Added some more methods for branches
af6ff9a

vinjai changed the title ~~Feature/write to branch~~Feature: Write to branchesJul 18, 2024

kevinjqliuand others added 11 commits July 29, 2024 08:39

s/"main"/MAIN_BRANCH
6fbf3f1
(cherry picked from commit ad1ec67)

replace string literals
8ce1509
(cherry picked from commit 40cf10e)

default writes to main branch
6daf29e
(cherry picked from commit 6dbe68d)

Added some more methods for branches
09321cd

Merged with master
60fef31

Updated antries for branches
45b01a6

Resolved Merge Conflict
917108b

Fixed some bugs
917b044

Fixed bugs in delete and overwrite
398f6c0

Added tests and some refactoring
b7b8ba0

Added another integration test
ee591b4

vinjai marked this pull request as ready for review October 16, 2024 08:15

Fixed bug: concurrent same name branch and tag writes
e81907d

kevinjqliu added this to the PyIceberg 0.9.0 release milestone Oct 30, 2024

kevinjqliu reviewed Nov 10, 2024
View reviewed changes

Comment threadtests/integration/test_writes/test_writes.py

vinjai added 2 commits November 14, 2024 04:07

Merge with main branch
4cf9198

Added integration tests with spark
bc6fb68

vinjai requested a review from kevinjqliu November 20, 2024 10:00

vinjai added 3 commits May 26, 2025 03:12

Merge branch 'main' into feature/write-to-branch
53a7f84

Fixed linter and code errors
e4463df

Fixed bug for empty tables
49f75b4

vinjai added 2 commits May 27, 2025 13:35

Fixed bugs and added more tests
4ed0607

changed design context for branch writes
958aac4

Merge branch 'main' into feature/write-to-branch
a0aae4d

Fokko approved these changes Jun 18, 2025
View reviewed changes

vinjai added 2 commits June 24, 2025 05:29

Merge branch 'main' into feature/write-to-branch
76249e9

Fixed linter, comments and other bugs
079802a

Usage of builder pattern
f45df8b

vinjai requested a review from Fokko June 28, 2025 06:29

Fokko approved these changes Jul 2, 2025
View reviewed changes

Fokko merged commit 33c8931 into apache:mainJul 2, 2025
10 checks passed

amitgilad3 pushed a commit to amitgilad3/iceberg-python that referenced this pull request Jul 7, 2025
Feature: Write to branches (apache#941)
3a37b52
Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

gabeiglio pushed a commit to Netflix/iceberg-python that referenced this pull request Aug 13, 2025
Feature: Write to branches (apache#941)
a9f82ca
Fixes: apache#306 --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

greenlaw mentioned this pull request Sep 4, 2025
Update add_files operation to support branches #2428
Closed

Uh oh!

Conversation

vinjai commented Jul 18, 2024

Uh oh!

vinjai commented Oct 16, 2024

Uh oh!

vinjai commented Oct 16, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu commented Oct 19, 2024

Uh oh!

vinjai commented Oct 19, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vinjai commented Nov 7, 2024

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vinjai commented Nov 14, 2024

Uh oh!

vinjai commented Nov 20, 2024

Uh oh!

vinjai commented May 23, 2025

Uh oh!

vinjai commented May 26, 2025

Uh oh!

vinjai commented May 27, 2025

Uh oh!

vinjai commented Jun 1, 2025

Uh oh!

Fokko commented Jun 1, 2025

Uh oh!

vinjai commented Jun 3, 2025

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

FokkoJun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented Jun 18, 2025

Uh oh!

SebastienN15 commented Jun 23, 2025

Uh oh!

vinjai commented Jun 24, 2025

Uh oh!

vinjai commented Jun 28, 2025

Uh oh!

Fokko commented Jul 2, 2025

Uh oh!

Uh oh!

jean-humann commented Jul 24, 2025

Uh oh!

kevinjqliu commented Jul 24, 2025

Uh oh!

averyrobbins commented Jul 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

vinjai commented Oct 16, 2024•
edited
Loading

vinjai commented Oct 19, 2024•
edited
Loading