Skip to content

feat(GCS+gRPC): more efficient CRC32C computations - #11070

Merged
coryan merged 1 commit into
googleapis:mainfrom
coryan:impl-GCS+gRPC-optimize-crc32c-computation-pr4
Mar 21, 2023
Merged

feat(GCS+gRPC): more efficient CRC32C computations#11070
coryan merged 1 commit into
googleapis:mainfrom
coryan:impl-GCS+gRPC-optimize-crc32c-computation-pr4

Conversation

@coryan

@coryancoryan commented Mar 20, 2023

Copy link
Copy Markdown
Contributor

Move the CRC32C and MD5 computations to the GrpcClient and RestClient. In the case of gRPC this allows us to compute the CRC32C checksum for each Write() message and then concatenate this checksum with the CRC32C checksum computed for the full object. That saves one of the most expensive computations in the gRPC uploads.

Fixes#11060


This change is Reviewable

Move the CRC32C and MD5 computations to the `GrpcClient` and
`RestClient`. In the case of gRPC this allows us to compute the CRC32C
checksum for each `Write()` message and then concatenate this checksum
with the CRC32C checksum computed for the full object. That saves one
of the most expensive computations in the gRPC uploads.
@codecov

codecovBot commented Mar 20, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 99.58% and project coverage change: -0.01⚠️

Comparison is base (cbd2f66) 93.76% compared to head (f9aad7c) 93.76%.

Additional details and impacted files
@@ Coverage Diff @@## main #11070 +/- ##
==========================================
- Coverage 93.76% 93.76% -0.01% 
==========================================
Files 1730 1730 Lines 156241 156319 +78 ==========================================
+ Hits 146504 146573 +69 - Misses 9737 9746 +9 
Impacted FilesCoverage Δ
...le/cloud/storage/internal/object_write_streambuf.h100.00% <ø> (ø)
google/cloud/storage/client.cc85.80% <66.66%> (+0.09%)⬆️
google/cloud/storage/hashing_options.cc100.00% <100.00%> (ø)
google/cloud/storage/idempotency_policy_test.cc100.00% <100.00%> (ø)
...d/storage/internal/async_accumulate_read_object.cc91.01% <100.00%> (ø)
google/cloud/storage/internal/curl_client.cc98.67% <100.00%> (ø)
google/cloud/storage/internal/curl_client_test.cc99.48% <100.00%> (ø)
google/cloud/storage/internal/grpc_client.cc98.89% <100.00%> (+0.45%)⬆️
google/cloud/storage/internal/grpc_client_test.cc100.00% <100.00%> (ø)
.../storage/internal/grpc_client_upload_chunk_test.cc100.00% <100.00%> (ø)
... and 9 more

... and 7 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@coryan
coryan marked this pull request as ready for review March 21, 2023 01:54
@coryan
coryan requested a review from a teamMarch 21, 2023 01:54
Comment threadgoogle/cloud/storage/internal/object_write_streambuf.cc
Comment threadgoogle/cloud/storage/internal/object_write_streambuf.cc
@coryan
coryan merged commit 6d07f6f into googleapis:mainMar 21, 2023
@coryan
coryan deleted the impl-GCS+gRPC-optimize-crc32c-computation-pr4 branch March 21, 2023 12:24
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize CRC32C computation in GCS+gRPC

2 participants

@coryan@devbww