Python: Add Google Cloud Storage support - #6906

Closed
Buktoria wants to merge 3 commits into
apache:masterfrom
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed

Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix

Conversation

@Buktoria

@BuktoriaBuktoria commented Feb 22, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Adding Google Cloud Storage support to the Python Iceberg CLI

  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)

@Buktoria
Buktoriaforce-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch from 68cc990 to 5345bb9CompareFebruary 25, 2023 01:42
@Fokko

Fokko commented Feb 27, 2023

Copy link
Copy Markdown
Contributor

@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!

@Buktoria
Buktoriaforce-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch 2 times, most recently from 2e59713 to aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria marked this pull request as ready for review March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch from aa58ac2 to 626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko reviewed Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile Outdated

test:
poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}

@FokkoFokkoMar 8, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged

@Buktoria
Buktoriaforce-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch 2 times, most recently from 9c5a826 to a3d430bCompareMarch 13, 2023 00:12
@Buktoria

Buktoria commented Mar 13, 2023

Copy link
Copy Markdown
ContributorAuthor

@Fokko This PR is ready to be reviewed

JonasJ-ap
JonasJ-ap reviewed Mar 22, 2023
View reviewed changes

@JonasJ-apJonasJ-ap left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!

Comment threadpython/pyiceberg/io/pyarrow.py
}
return GcsFileSystem(**client_kwargs)
else:
return GcsFileSystem()

@JonasJ-apJonasJ-apMar 22, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:

elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)

?
Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.

@Fokko
Fokko self-requested a review April 10, 2023 22:28
@Fokko

Fokko commented Apr 10, 2023

Copy link
Copy Markdown
Contributor

@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.

@FokkoFokko added this to the PyIceberg 0.4.0 release milestone Apr 18, 2023
Fokko
Fokko reviewed Apr 24, 2023
View reviewed changes

@FokkoFokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0

Comment threadpython/pyiceberg/io/fsspec.py Outdated
@FokkoFokko changed the title pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko

Fokko commented May 14, 2023

Copy link
Copy Markdown
Contributor

@Buktoria Gentle ping! :)

@Buktoria

Buktoria commented May 23, 2023

Copy link
Copy Markdown
ContributorAuthor

Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.

@Fokko

Fokko commented Jun 13, 2023

Copy link
Copy Markdown
Contributor

@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.

@Fokko

Fokko commented Jul 18, 2023

Copy link
Copy Markdown
Contributor

@Buktoria Gente ping from my side. Would be great to get this in.

danielcweeks
danielcweeks reviewed Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py Outdated
}
return S3FileSystem(**client_kwargs)
elif scheme in {"gs", "gcs"}:
if access_token := self.properties.get("gs.access-token"):

@danielcweeksdanielcweeksJul 18, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme

danielcweeks
danielcweeks reviewed Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py Outdated
}
return S3FileSystem(**client_kwargs)
elif scheme in {"gs", "gcs"}:
if access_token := self.properties.get("gs.access-token"):

@danielcweeksdanielcweeksJul 18, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?

@BuktoriaBuktoriaJul 24, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.

bryanck
bryanck reviewed Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage

@bryanckbryanckJul 19, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.

@Buktoria
Buktoriaforce-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch 2 times, most recently from 71e7135 to d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch from d17d112 to 8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko reviewed Jul 24, 2023
View reviewed changes

@FokkoFokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?

Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?

Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |

@FokkoFokkoJul 24, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |

@FokkoFokko mentioned this pull request Aug 2, 2023
Merged
@Fokko

Fokko commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻

@FokkoFokko closed this Aug 14, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments
+1 more reviewer
@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

python

Projects

None yet

Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Python: Add Google Cloud Storage support - #6906
Closed
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed
Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Conversation
@Buktoria
@BuktoriaBuktoria
commented
Feb 22, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Adding Google Cloud Storage support to the Python Iceberg CLI
  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
68cc990 to
5345bb9CompareFebruary 25, 2023 01:42
@Fokko
Fokko
commented
Feb 27, 2023

Copy link


Copy Markdown

Contributor
@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
2e59713 to
aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria
marked this pull request as ready for review
March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
aa58ac2 to
626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko
reviewed
Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile
Outdated


test:
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}
@FokkoFokkoMar 8, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
9c5a826 to
a3d430bCompareMarch 13, 2023 00:12
@Buktoria
Buktoria
commented
Mar 13, 2023

Copy link


Copy Markdown

ContributorAuthor
@Fokko This PR is ready to be reviewed
JonasJ-ap
JonasJ-ap
reviewed
Mar 22, 2023
View reviewed changes
@JonasJ-apJonasJ-ap
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!
Comment threadpython/pyiceberg/io/pyarrow.py
 }
 return GcsFileSystem(**client_kwargs)
 else:
 return GcsFileSystem()
@JonasJ-apJonasJ-apMar 22, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:
elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)
?

Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.
@Fokko
Fokko
self-requested a review
April 10, 2023 22:28
@Fokko
Fokko
commented
Apr 10, 2023

Copy link


Copy Markdown

Contributor
@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.
@FokkoFokko
added this to the PyIceberg 0.4.0 release milestone
Apr 18, 2023
Fokko
Fokko
reviewed
Apr 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0
Comment threadpython/pyiceberg/io/fsspec.py
Outdated

@FokkoFokko
changed the title
pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko
Fokko
commented
May 14, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gentle ping! :)
@Buktoria
Buktoria
commented
May 23, 2023

Copy link


Copy Markdown

ContributorAuthor
Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.
@Fokko
Fokko
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.
@Fokko
Fokko
commented
Jul 18, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gente ping from my side. Would be great to get this in.
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?
@BuktoriaBuktoriaJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.
bryanck
bryanck
reviewed
Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage
@bryanckbryanckJul 19, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
71e7135 to
d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
d17d112 to
8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko
reviewed
Jul 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?
Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?
Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |
@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |
@FokkoFokko
mentioned this pull request
Aug 2, 2023
 Merged

@Fokko
Fokko
commented
Aug 14, 2023

Copy link


Copy Markdown

Contributor
Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻
@FokkoFokko
closed this
Aug 14, 2023
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments

+1 more reviewer

@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

python

Projects


None yet


Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Python: Add Google Cloud Storage support - #6906
Closed
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed
Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Conversation
@Buktoria
@BuktoriaBuktoria
commented
Feb 22, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Adding Google Cloud Storage support to the Python Iceberg CLI
  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
68cc990 to
5345bb9CompareFebruary 25, 2023 01:42
@Fokko
Fokko
commented
Feb 27, 2023

Copy link


Copy Markdown

Contributor
@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
2e59713 to
aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria
marked this pull request as ready for review
March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
aa58ac2 to
626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko
reviewed
Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile
Outdated


test:
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}
@FokkoFokkoMar 8, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
9c5a826 to
a3d430bCompareMarch 13, 2023 00:12
@Buktoria
Buktoria
commented
Mar 13, 2023

Copy link


Copy Markdown

ContributorAuthor
@Fokko This PR is ready to be reviewed
JonasJ-ap
JonasJ-ap
reviewed
Mar 22, 2023
View reviewed changes
@JonasJ-apJonasJ-ap
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!
Comment threadpython/pyiceberg/io/pyarrow.py
 }
 return GcsFileSystem(**client_kwargs)
 else:
 return GcsFileSystem()
@JonasJ-apJonasJ-apMar 22, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:
elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)
?

Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.
@Fokko
Fokko
self-requested a review
April 10, 2023 22:28
@Fokko
Fokko
commented
Apr 10, 2023

Copy link


Copy Markdown

Contributor
@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.
@FokkoFokko
added this to the PyIceberg 0.4.0 release milestone
Apr 18, 2023
Fokko
Fokko
reviewed
Apr 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0
Comment threadpython/pyiceberg/io/fsspec.py
Outdated

@FokkoFokko
changed the title
pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko
Fokko
commented
May 14, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gentle ping! :)
@Buktoria
Buktoria
commented
May 23, 2023

Copy link


Copy Markdown

ContributorAuthor
Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.
@Fokko
Fokko
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.
@Fokko
Fokko
commented
Jul 18, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gente ping from my side. Would be great to get this in.
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?
@BuktoriaBuktoriaJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.
bryanck
bryanck
reviewed
Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage
@bryanckbryanckJul 19, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
71e7135 to
d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
d17d112 to
8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko
reviewed
Jul 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?
Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?
Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |
@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |
@FokkoFokko
mentioned this pull request
Aug 2, 2023
 Merged

@Fokko
Fokko
commented
Aug 14, 2023

Copy link


Copy Markdown

Contributor
Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻
@FokkoFokko
closed this
Aug 14, 2023
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments

+1 more reviewer

@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

python

Projects


None yet


Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Python: Add Google Cloud Storage support - #6906
Closed
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed
Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Conversation
@Buktoria
@BuktoriaBuktoria
commented
Feb 22, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Adding Google Cloud Storage support to the Python Iceberg CLI
  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
68cc990 to
5345bb9CompareFebruary 25, 2023 01:42
@Fokko
Fokko
commented
Feb 27, 2023

Copy link


Copy Markdown

Contributor
@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
2e59713 to
aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria
marked this pull request as ready for review
March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
aa58ac2 to
626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko
reviewed
Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile
Outdated


test:
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}
@FokkoFokkoMar 8, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
9c5a826 to
a3d430bCompareMarch 13, 2023 00:12
@Buktoria
Buktoria
commented
Mar 13, 2023

Copy link


Copy Markdown

ContributorAuthor
@Fokko This PR is ready to be reviewed
JonasJ-ap
JonasJ-ap
reviewed
Mar 22, 2023
View reviewed changes
@JonasJ-apJonasJ-ap
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!
Comment threadpython/pyiceberg/io/pyarrow.py
 }
 return GcsFileSystem(**client_kwargs)
 else:
 return GcsFileSystem()
@JonasJ-apJonasJ-apMar 22, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:
elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)
?

Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.
@Fokko
Fokko
self-requested a review
April 10, 2023 22:28
@Fokko
Fokko
commented
Apr 10, 2023

Copy link


Copy Markdown

Contributor
@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.
@FokkoFokko
added this to the PyIceberg 0.4.0 release milestone
Apr 18, 2023
Fokko
Fokko
reviewed
Apr 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0
Comment threadpython/pyiceberg/io/fsspec.py
Outdated

@FokkoFokko
changed the title
pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko
Fokko
commented
May 14, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gentle ping! :)
@Buktoria
Buktoria
commented
May 23, 2023

Copy link


Copy Markdown

ContributorAuthor
Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.
@Fokko
Fokko
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.
@Fokko
Fokko
commented
Jul 18, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gente ping from my side. Would be great to get this in.
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?
@BuktoriaBuktoriaJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.
bryanck
bryanck
reviewed
Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage
@bryanckbryanckJul 19, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
71e7135 to
d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
d17d112 to
8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko
reviewed
Jul 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?
Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?
Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |
@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |
@FokkoFokko
mentioned this pull request
Aug 2, 2023
 Merged

@Fokko
Fokko
commented
Aug 14, 2023

Copy link


Copy Markdown

Contributor
Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻
@FokkoFokko
closed this
Aug 14, 2023
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments

+1 more reviewer

@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

python

Projects


None yet


Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Python: Add Google Cloud Storage support - #6906
Closed
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed
Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Conversation
@Buktoria
@BuktoriaBuktoria
commented
Feb 22, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Adding Google Cloud Storage support to the Python Iceberg CLI
  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
68cc990 to
5345bb9CompareFebruary 25, 2023 01:42
@Fokko
Fokko
commented
Feb 27, 2023

Copy link


Copy Markdown

Contributor
@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
2e59713 to
aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria
marked this pull request as ready for review
March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
aa58ac2 to
626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko
reviewed
Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile
Outdated


test:
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}
@FokkoFokkoMar 8, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
9c5a826 to
a3d430bCompareMarch 13, 2023 00:12
@Buktoria
Buktoria
commented
Mar 13, 2023

Copy link


Copy Markdown

ContributorAuthor
@Fokko This PR is ready to be reviewed
JonasJ-ap
JonasJ-ap
reviewed
Mar 22, 2023
View reviewed changes
@JonasJ-apJonasJ-ap
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!
Comment threadpython/pyiceberg/io/pyarrow.py
 }
 return GcsFileSystem(**client_kwargs)
 else:
 return GcsFileSystem()
@JonasJ-apJonasJ-apMar 22, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:
elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)
?

Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.
@Fokko
Fokko
self-requested a review
April 10, 2023 22:28
@Fokko
Fokko
commented
Apr 10, 2023

Copy link


Copy Markdown

Contributor
@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.
@FokkoFokko
added this to the PyIceberg 0.4.0 release milestone
Apr 18, 2023
Fokko
Fokko
reviewed
Apr 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0
Comment threadpython/pyiceberg/io/fsspec.py
Outdated

@FokkoFokko
changed the title
pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko
Fokko
commented
May 14, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gentle ping! :)
@Buktoria
Buktoria
commented
May 23, 2023

Copy link


Copy Markdown

ContributorAuthor
Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.
@Fokko
Fokko
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.
@Fokko
Fokko
commented
Jul 18, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gente ping from my side. Would be great to get this in.
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?
@BuktoriaBuktoriaJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.
bryanck
bryanck
reviewed
Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage
@bryanckbryanckJul 19, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
71e7135 to
d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
d17d112 to
8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko
reviewed
Jul 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?
Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?
Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |
@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |
@FokkoFokko
mentioned this pull request
Aug 2, 2023
 Merged

@Fokko
Fokko
commented
Aug 14, 2023

Copy link


Copy Markdown

Contributor
Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻
@FokkoFokko
closed this
Aug 14, 2023
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments

+1 more reviewer

@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

python

Projects


None yet


Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Python: Add Google Cloud Storage support - #6906
Closed
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed
Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Conversation
@Buktoria
@BuktoriaBuktoria
commented
Feb 22, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Adding Google Cloud Storage support to the Python Iceberg CLI
  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
68cc990 to
5345bb9CompareFebruary 25, 2023 01:42
@Fokko
Fokko
commented
Feb 27, 2023

Copy link


Copy Markdown

Contributor
@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
2e59713 to
aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria
marked this pull request as ready for review
March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
aa58ac2 to
626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko
reviewed
Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile
Outdated


test:
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}
@FokkoFokkoMar 8, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
9c5a826 to
a3d430bCompareMarch 13, 2023 00:12
@Buktoria
Buktoria
commented
Mar 13, 2023

Copy link


Copy Markdown

ContributorAuthor
@Fokko This PR is ready to be reviewed
JonasJ-ap
JonasJ-ap
reviewed
Mar 22, 2023
View reviewed changes
@JonasJ-apJonasJ-ap
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!
Comment threadpython/pyiceberg/io/pyarrow.py
 }
 return GcsFileSystem(**client_kwargs)
 else:
 return GcsFileSystem()
@JonasJ-apJonasJ-apMar 22, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:
elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)
?

Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.
@Fokko
Fokko
self-requested a review
April 10, 2023 22:28
@Fokko
Fokko
commented
Apr 10, 2023

Copy link


Copy Markdown

Contributor
@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.
@FokkoFokko
added this to the PyIceberg 0.4.0 release milestone
Apr 18, 2023
Fokko
Fokko
reviewed
Apr 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0
Comment threadpython/pyiceberg/io/fsspec.py
Outdated

@FokkoFokko
changed the title
pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko
Fokko
commented
May 14, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gentle ping! :)
@Buktoria
Buktoria
commented
May 23, 2023

Copy link


Copy Markdown

ContributorAuthor
Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.
@Fokko
Fokko
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.
@Fokko
Fokko
commented
Jul 18, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gente ping from my side. Would be great to get this in.
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?
@BuktoriaBuktoriaJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.
bryanck
bryanck
reviewed
Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage
@bryanckbryanckJul 19, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
71e7135 to
d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
d17d112 to
8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko
reviewed
Jul 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?
Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?
Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |
@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |
@FokkoFokko
mentioned this pull request
Aug 2, 2023
 Merged

@Fokko
Fokko
commented
Aug 14, 2023

Copy link


Copy Markdown

Contributor
Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻
@FokkoFokko
closed this
Aug 14, 2023
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments

+1 more reviewer

@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

python

Projects


None yet


Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Python: Add Google Cloud Storage support - #6906
Closed
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed
Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Conversation
@Buktoria
@BuktoriaBuktoria
commented
Feb 22, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Adding Google Cloud Storage support to the Python Iceberg CLI
  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
68cc990 to
5345bb9CompareFebruary 25, 2023 01:42
@Fokko
Fokko
commented
Feb 27, 2023

Copy link


Copy Markdown

Contributor
@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
2e59713 to
aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria
marked this pull request as ready for review
March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
aa58ac2 to
626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko
reviewed
Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile
Outdated


test:
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}
@FokkoFokkoMar 8, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
9c5a826 to
a3d430bCompareMarch 13, 2023 00:12
@Buktoria
Buktoria
commented
Mar 13, 2023

Copy link


Copy Markdown

ContributorAuthor
@Fokko This PR is ready to be reviewed
JonasJ-ap
JonasJ-ap
reviewed
Mar 22, 2023
View reviewed changes
@JonasJ-apJonasJ-ap
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!
Comment threadpython/pyiceberg/io/pyarrow.py
 }
 return GcsFileSystem(**client_kwargs)
 else:
 return GcsFileSystem()
@JonasJ-apJonasJ-apMar 22, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:
elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)
?

Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.
@Fokko
Fokko
self-requested a review
April 10, 2023 22:28
@Fokko
Fokko
commented
Apr 10, 2023

Copy link


Copy Markdown

Contributor
@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.
@FokkoFokko
added this to the PyIceberg 0.4.0 release milestone
Apr 18, 2023
Fokko
Fokko
reviewed
Apr 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0
Comment threadpython/pyiceberg/io/fsspec.py
Outdated

@FokkoFokko
changed the title
pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko
Fokko
commented
May 14, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gentle ping! :)
@Buktoria
Buktoria
commented
May 23, 2023

Copy link


Copy Markdown

ContributorAuthor
Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.
@Fokko
Fokko
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.
@Fokko
Fokko
commented
Jul 18, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gente ping from my side. Would be great to get this in.
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?
@BuktoriaBuktoriaJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.
bryanck
bryanck
reviewed
Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage
@bryanckbryanckJul 19, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
71e7135 to
d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
d17d112 to
8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko
reviewed
Jul 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?
Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?
Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |
@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |
@FokkoFokko
mentioned this pull request
Aug 2, 2023
 Merged

@Fokko
Fokko
commented
Aug 14, 2023

Copy link


Copy Markdown

Contributor
Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻
@FokkoFokko
closed this
Aug 14, 2023
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments

+1 more reviewer

@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

python

Projects


None yet


Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Python: Add Google Cloud Storage support - #6906
Closed
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Closed
Python: Add Google Cloud Storage support#6906
Buktoria wants to merge 3 commits into
apache:masterfrom 
Buktoria:pyiceberg/accpet-file-paths-with-gs-prefix
Conversation
@Buktoria
@BuktoriaBuktoria
commented
Feb 22, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Adding Google Cloud Storage support to the Python Iceberg CLI
  • Added GCSFileIO support through the gcsfs python lib
  • Added PyArrow GCS support with PyArrows GcsFileSystem class
  • Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
  • Updated docs to include GCSFileIO configs options (full list of possible options can be found here)
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
68cc990 to
5345bb9CompareFebruary 25, 2023 01:42
@Fokko
Fokko
commented
Feb 27, 2023

Copy link


Copy Markdown

Contributor
@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
2e59713 to
aa58ac2CompareMarch 8, 2023 01:48
@Buktoria
Buktoria
marked this pull request as ready for review
March 8, 2023 01:59
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
aa58ac2 to
626aa7eCompareMarch 8, 2023 02:33
Fokko
Fokko
reviewed
Mar 8, 2023
View reviewed changes
Comment threadpython/Makefile
Outdated


test:
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}
@FokkoFokkoMar 8, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get prettier once #6398 has been merged
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
9c5a826 to
a3d430bCompareMarch 13, 2023 00:12
@Buktoria
Buktoria
commented
Mar 13, 2023

Copy link


Copy Markdown

ContributorAuthor
@Fokko This PR is ready to be reviewed
JonasJ-ap
JonasJ-ap
reviewed
Mar 22, 2023
View reviewed changes
@JonasJ-apJonasJ-ap
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your great work!
Comment threadpython/pyiceberg/io/pyarrow.py
 }
 return GcsFileSystem(**client_kwargs)
 else:
 return GcsFileSystem()
@JonasJ-apJonasJ-apMar 22, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] Is it possible to take out this if...else..., like:
elifschemein {"gs", "gcs"}:
client_kwargs= {
"access_token": self.properties.get("gs.access-token"),
"credential_token_expiration": self.properties.get("gs.credential-token-expiration"),
}
returnGcsFileSystem(**client_kwargs)
?

Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.
@Fokko
Fokko
self-requested a review
April 10, 2023 22:28
@Fokko
Fokko
commented
Apr 10, 2023

Copy link


Copy Markdown

Contributor
@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.
@FokkoFokko
added this to the PyIceberg 0.4.0 release milestone
Apr 18, 2023
Fokko
Fokko
reviewed
Apr 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0
Comment threadpython/pyiceberg/io/fsspec.py
Outdated

@FokkoFokko
changed the title
pyiceberg: Add Google Cloud Storage supportPython: Add Google Cloud Storage supportApr 24, 2023
@Fokko
Fokko
commented
May 14, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gentle ping! :)
@Buktoria
Buktoria
commented
May 23, 2023

Copy link


Copy Markdown

ContributorAuthor
Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.
@Fokko
Fokko
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.
@Fokko
Fokko
commented
Jul 18, 2023

Copy link


Copy Markdown

Contributor
@Buktoria Gente ping from my side. Would be great to get this in.
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this property should be gs.token based on the properties in the readme
danielcweeks
danielcweeks
reviewed
Jul 18, 2023
View reviewed changes
Comment threadpython/pyiceberg/io/pyarrow.py
Outdated

 }
 return S3FileSystem(**client_kwargs)
 elif scheme in {"gs", "gcs"}:
 if access_token := self.properties.get("gs.access-token"):
@danielcweeksdanielcweeksJul 18, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add gs.credential-token-expiration to the docs?
@BuktoriaBuktoriaJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed it. I've added it to the docs now.
bryanck
bryanck
reviewed
Jul 19, 2023
View reviewed changes
Comment threadpython/mkdocs/docs/configuration.md
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |

### Google Cloud Storage
@bryanckbryanckJul 19, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
2 times, most recently
from
71e7135 to
d17d112CompareJuly 24, 2023 00:05
@Buktoria
Buktoriaforce-pushed
the
pyiceberg/accpet-file-paths-with-gs-prefix
branch
from
d17d112 to
8b2ba33CompareJuly 24, 2023 00:08
@Buktoria@Fokko
Update python/pyiceberg/io/fsspec.py
3740c69
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Fokko
Fokko
reviewed
Jul 24, 2023
View reviewed changes
@FokkoFokko
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?
Comment threadpython/mkdocs/docs/configuration.md
| Key | Example | Description |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |

@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this unrelated change?
Comment threadpython/mkdocs/docs/index.md
| s3fs | S3FS as a FileIO implementation to interact with the object store |
| adlfs | ADLFS as a FileIO implementation to interact with the object store |
| snappy | Support for snappy Avro compression |
| gcs | GCS as the FileIO implementation to interact with the object store |
@FokkoFokkoJul 24, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

|gcs | GCS as the FileIO implementation to interact with the object store |
|gcsfs| GCS as the FileIO implementation to interact with the object store |
@FokkoFokko
mentioned this pull request
Aug 2, 2023
 Merged

@Fokko
Fokko
commented
Aug 14, 2023

Copy link


Copy Markdown

Contributor
Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻
@FokkoFokko
closed this
Aug 14, 2023
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@FokkoFokkoFokko left review comments
@danielcweeksdanielcweeksdanielcweeks left review comments
@bryanckbryanckbryanck left review comments

+1 more reviewer

@JonasJ-apJonasJ-apJonasJ-ap left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

python

Projects


None yet


Milestone

PyIceberg 0.5.0 release

Development

Successfully merging this pull request may close these issues.

5 participants

@Buktoria@Fokko@danielcweeks@bryanck@JonasJ-ap