Skip to content
This repository was archived by the owner on Mar 31, 2026. It is now read-only.

Feat: Add "auto" checksum option and make default - #1383

Merged
andrewsg merged 5 commits into
3.0-develfrom
3.0-checksum-defaults
Nov 21, 2024
Merged

Feat: Add "auto" checksum option and make default#1383
andrewsg merged 5 commits into
3.0-develfrom
3.0-checksum-defaults

Conversation

@andrewsg

Copy link
Copy Markdown
Contributor

No description provided.

@andrewsg
andrewsg requested review from a teamNovember 16, 2024 06:03
@product-auto-labelproduct-auto-labelBot added size: l Pull request size is large. api: storage Issues related to the googleapis/python-storage API. labels Nov 16, 2024
@andrewsg
andrewsgforce-pushed the 3.0-checksum-defaults branch from 7c76ff3 to a82d9b8CompareNovember 16, 2024 06:11
@andrewsg
andrewsgforce-pushed the 3.0-checksum-defaults branch from a82d9b8 to 25e1109CompareNovember 18, 2024 18:18

@cojencocojenco left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding clear documentation too! Just a question on adding a parametrized checksum "auto" to the system test.

Comment threadREADME.rst
downloads and None for most uploads. Note that ranged downloads ("start" or
"end" set) still do not support any checksumming, and some features in
`transfer_manager.py` still support crc32c only.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this! This will make 3.0 release notes really clear

if self.checksum == "auto":
self.checksum = (
"crc32c" if _helpers._is_crc32c_available_and_fast() else "md5"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like how this is handled in the constructor 🎉

Comment threadgoogle/cloud/storage/_media/_helpers.py
assert upload.upload_url == MULTIPART_URL
assert upload._headers == {}
assert upload._checksum_type is None
assert upload._checksum_type == "crc32c" # converted from "auto"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice 😄


EMPTY_MD5 = base64.b64encode(hashlib.md5(b"").digest()).decode("utf-8")
crc32c = _helpers._get_crc32c_object()
crc32c = google_crc32c.Checksum()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For system tests, maybe we can add a parametrized checksum "auto" here?

@pytest.mark.parametrize("checksum", ["md5", "crc32c", None])
deftest_download_full(self, add_files, authorized_transport, checksum):

Can do the same for an upload integration test here as well

@pytest.mark.parametrize("checksum", ["md5", "crc32c", None])
deftest_multipart_upload(authorized_transport, bucket, cleanup, checksum):

@andrewsg
andrewsgforce-pushed the 3.0-checksum-defaults branch from b2bb316 to c4201c6CompareNovember 21, 2024 01:38
@andrewsg
andrewsg merged commit 8d32f25 into 3.0-develNov 21, 2024
@andrewsg
andrewsg deleted the 3.0-checksum-defaults branch November 21, 2024 01:53
@release-pleaserelease-pleaseBot mentioned this pull request Dec 11, 2024
@andrewsgandrewsg mentioned this pull request Dec 12, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

api: storageIssues related to the googleapis/python-storage API.size: lPull request size is large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@andrewsg@cojenco