Moving MD5 hash method from regression to storage helpers. by dhermes · Pull Request #611 · googleapis/google-cloud-python

dhermes · 2015-02-10T07:45:50Z

May make sense to swap out for hashlib.md5 (though they seem to
be equivalent).

In preparation for #547.

dhermes · 2015-02-10T07:51:38Z

If we really want to do this, we'll need to support computing MD5 of files that don't fit in memory:
http://stackoverflow.com/a/1131255/1068170

This is done in a generic way in gsutil (though the method is quite a mouthful).

dhermes · 2015-02-10T07:54:27Z

@tseaver The lint failure is a bug in my PR speedup code since among

gcloud/storage/_helpers.py
gcloud/storage/test__helpers.py
regression/storage.py

the abstract class is really never used.

We have 3 options:

~~I make a temporary commit disabling the PR speedup to just diff against all the files~~
I try to find a full-time fix (done in Repeating pylint after failures. #612)
~~Ignore and make sure it passes when merged~~

This surfaced as a false negative in googleapis#611.

tseaver · 2015-02-10T21:11:53Z

LGTM. I'd say we should probably not set the envvars on Travis, to avoid these kinds of errors.

dhermes · 2015-02-10T21:14:23Z

We don't set the env vars on merges to master. This only happens for PRs, which we want to be quick and snappy during review.

Sign in to view

dhermes · 2015-02-12T06:04:41Z

Went with 8192 for the chunk size since gsutil does:
https://github.com/GoogleCloudPlatform/gsutil/blob/bc083d405b92856a1aba46a65410bd1a1dbf65a6/gslib/util.py#L66

@tseaver PTAL

I also made a separate method for updating a hash object for when CRC32 comes around. (Sort of YAGNI, sort of not.)

tseaver · 2015-02-12T18:45:18Z

@dhermes pylint failure due to partial-set checking.

dhermes · 2015-02-12T18:46:07Z

Yes, recall that this PR is the reason for #612.

May make sense to swap out for `hashlib.md5` (though they seem to be equivalent). In preparation for googleapis#547.

This is to account for files too large to fit into memory.

dhermes · 2015-02-12T18:53:31Z

@tseaver I just rebased on top of #612.

coveralls · 2015-02-12T18:57:11Z

Coverage remained the same at 100.0% when pulling 86b1f59 on dhermes:move-md5-to-library into df4c625 on GoogleCloudPlatform:master.

tseaver · 2015-02-12T19:23:41Z

LGTM.

Moving MD5 hash method from regression to storage helpers.

dhermes · 2015-02-12T19:32:25Z

It seems coveralls has recovered from whatever was making it silently fail last night.

* docs: Add documentation for enums fix: Add context manager return types chore: Update gapic-generator-python to v1.8.1 PiperOrigin-RevId: 503210727 Source-Link: googleapis/googleapis@a391fd1 Source-Link: googleapis/googleapis-gen@0080f83 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDA4MGY4MzBkZWMzN2MzMzg0MTU3MDgyYmNlMjc5ZTM3MDc5ZWE1OCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

Source-Link: googleapis/synthtool@69fabae Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:562802bfac02e012a6ac34eda282f81d06e77326b82a32d7bbb1369ff552b387 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Add unit tests for many spanner_django modules that add coverage beyond the built-in django tests.

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Feb 10, 2015

dhermes added a commit to dhermes/google-cloud-python that referenced this pull request Feb 10, 2015
Repeating pylint after failures.
b8d3caf
This surfaced as a false negative in googleapis#611.

dhermes mentioned this pull request Feb 10, 2015
Repeating pylint after failures. #612
Merged

tseaver reviewed Feb 11, 2015
View reviewed changes

Comment threadgcloud/storage/_helpers.py Outdated
This comment was marked as spam.
Sign in to view
This comment was marked as spam.
Sign in to view

dhermes added 2 commits February 12, 2015 10:51

Moving MD5 hash method from regression to storage helpers.
e062c89
May make sense to swap out for `hashlib.md5` (though they seem to be equivalent). In preparation for googleapis#547.

Making MD5 hash method take a buffer instead of bytes.
86b1f59
This is to account for files too large to fit into memory.

dhermes force-pushed the move-md5-to-library branch from 06526ad to 86b1f59Compare February 12, 2015 18:53

dhermes added a commit that referenced this pull request Feb 12, 2015
Merge pull request#611from dhermes/move-md5-to-library
c88ba72
Moving MD5 hash method from regression to storage helpers.

dhermes merged commit c88ba72 into googleapis:masterFeb 12, 2015

dhermes deleted the move-md5-to-library branch February 12, 2015 19:32

dhermes mentioned this pull request Feb 14, 2015
Upgrading version to 0.4.1. #640
Merged

dhermes mentioned this pull request Mar 9, 2015
Adding _base64_crc32c function for storage integrity checks. #692
Closed

parthea pushed a commit that referenced this pull request Aug 21, 2025
chore(deps): update all dependencies (#611)
287bd85

parthea pushed a commit that referenced this pull request Sep 16, 2025
chore(deps): update all dependencies (#611)
30da01e

parthea pushed a commit that referenced this pull request Nov 22, 2025
chore(main): release 2.11.0 (#611)
a52f0f9
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

parthea pushed a commit that referenced this pull request Mar 2, 2026
chore(deps): update dependency google-cloud-pubsub to v2.11.0 (#611)
91bd83c

parthea pushed a commit that referenced this pull request Mar 6, 2026
chore: prevent normalization of semver versioning (#611)
1762f29

parthea pushed a commit that referenced this pull request Mar 6, 2026
feat: added unit test with coverage of 68% (#611)
11fc186
Add unit tests for many spanner_django modules that add coverage beyond the built-in django tests.

Navigation Menu

Uh oh!

Moving MD5 hash method from regression to storage helpers. - #611

Moving MD5 hash method from regression to storage helpers.#611
dhermes merged 2 commits into
googleapis:masterfrom
dhermes:move-md5-to-library

dhermes commented Feb 10, 2015

Uh oh!

dhermes commented Feb 10, 2015

Uh oh!

dhermes commented Feb 10, 2015

Uh oh!

tseaver commented Feb 10, 2015

Uh oh!

dhermes commented Feb 10, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

tseaver commented Feb 12, 2015

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

coveralls commented Feb 12, 2015

Uh oh!

tseaver commented Feb 12, 2015

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

dhermes commented Feb 10, 2015

Uh oh!

dhermes commented Feb 10, 2015

Uh oh!

dhermes commented Feb 10, 2015

Uh oh!

tseaver commented Feb 10, 2015

Uh oh!

dhermes commented Feb 10, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

tseaver commented Feb 12, 2015

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

coveralls commented Feb 12, 2015

Uh oh!

tseaver commented Feb 12, 2015

Uh oh!

dhermes commented Feb 12, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants