Skip to content

impl(storage): new helpers for CRC32C computation - #11061

Merged
coryan merged 3 commits into
googleapis:mainfrom
coryan:impl-GCS+gRPC-optimize-crc32c-computation-pr2
Mar 18, 2023
Merged

impl(storage): new helpers for CRC32C computation#11061
coryan merged 3 commits into
googleapis:mainfrom
coryan:impl-GCS+gRPC-optimize-crc32c-computation-pr2

Conversation

@coryan

@coryancoryan commented Mar 17, 2023

Copy link
Copy Markdown
Contributor

I am adding new functions to compute CRC32C checksums. When using recent versions of Abseil (>= 20230125) these functions can concatenate the checksum of two buffers. I am planning to use this in the GCS+gRPC plugin, where each Write() message includes a CRC32C and we also need to send the end-to-end checksum of all the Write() messages.

Part of the work #11060


This change is Reviewable

I am adding new functions to compute CRC32C checksums. When using
recent versions of Abseil (>= 20230125) these functions can concatenate
the checksum of two buffers. I am planning to use this in the GCS+gRPC
plugin, where each `Write()` message includes a CRC32C and we also need
to send the end-to-end checksum of all the `Write()` messages.
@product-auto-labelproduct-auto-labelBot added the api: storage Issues related to the Cloud Storage API. label Mar 17, 2023
@codecov

codecovBot commented Mar 17, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (afeb258) 93.75% compared to head (3542d85) 93.76%.

Additional details and impacted files
@@ Coverage Diff @@## main #11061 +/- ##
========================================
Coverage 93.75% 93.76% ========================================
Files 1727 1730 +3 Lines 155798 155911 +113 ========================================
+ Hits 146069 146186 +117 + Misses 9729 9725 -4 
Impacted FilesCoverage Δ
google/cloud/storage/internal/crc32c.cc100.00% <100.00%> (ø)
google/cloud/storage/internal/crc32c.h100.00% <100.00%> (ø)
google/cloud/storage/internal/crc32c_test.cc100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@coryan
coryan marked this pull request as ready for review March 17, 2023 20:28
@coryan
coryan requested a review from a teamMarch 17, 2023 20:28
Comment threadgoogle/cloud/storage/BUILD.bazel
Comment threadgoogle/cloud/storage/google_cloud_cpp_storage.cmake
Comment threadgoogle/cloud/storage/internal/crc32c_benchmark.cc
@coryan
coryan merged commit cc2a564 into googleapis:mainMar 18, 2023
@coryan
coryan deleted the impl-GCS+gRPC-optimize-crc32c-computation-pr2 branch March 18, 2023 17:48
kou added a commit to kou/google-cloud-cpp that referenced this pull request Aug 25, 2025
cc2a564 (googleapis#11061) introduced Abseil
based CRC32C implementation. So we don't require crc32c when we use
recent Abseil.
If we don't require crc32c with recent Abseil, downstream users such
as Apache Arrow don't need to bundle crc32c. It reduces maintenance
cost.
See also: apache/arrow#45989
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storageIssues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@coryan@dbolduc