zlib: expose zlib.crc32() by joyeecheung · Pull Request #52692 · nodejs/node

joyeecheung · 2024-04-25T17:53:00Z

This patch exposes the crc32() function from zlib to user-land.

It computes a 32-bit Cyclic Redundancy Check checksum of data. If
value is specified, it is used as the starting value of the checksum,
otherwise, 0 is used as the starting value.

The CRC algorithm is designed to compute checksums and to detect error
in data transmission. It's not suitable for cryptographic authentication.

To be consistent with other APIs, if the data is a string, it will
be encoded with UTF-8 before being used for computation. If users only
use Node.js to compute and match the checksums, this works well with
other APIs that uses the UTF-8 encoding by default.

Some third-party JavaScript libraries compute the checksum on a
string based on str.charCodeAt() so that it can be run in browsers.
If users want to match the checksum computed with this kind of library
in the browser, it's better to use the same library in Node.js
if it also runs in Node.js. If users have to use zlib.crc32() to
match the checksum produced by such a third-party library:

If the library accepts Uint8Array as input, use TextEncoder
in the browser to encode the string into a Uint8Array with UTF-8
encoding, and compute the checksum based on the UTF-8 encoded string
in the browser.
If the library only takes a string and compute the data based on
str.charCodeAt(), on the Node.js side, convert the string into
a buffer using Buffer.from(str, 'utf16le').

constzlib=require('node:zlib');const{ Buffer }=require('node:buffer');letcrc=zlib.crc32('hello');// 907060870crc=zlib.crc32('world',crc);// 4192936109crc=zlib.crc32(Buffer.from('hello','utf16le'));// 1427272415crc=zlib.crc32(Buffer.from('world','utf16le'),crc);// 4150509955

NOTE: it's exposed in zlib because that's also what Python and Ruby do.

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ```

lpinca · 2024-04-25T19:33:08Z

Wouldn't it be better to expose it under crypto?

Edit: On second thought, maybe not.

joyeecheung · 2024-04-25T19:54:39Z

Wouldn't it be better to expose it under crypto?

It's not a cryptographic hash, but a checksum. I was thinking about copying the python documentation (python also puts it on zlib):

The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm.

(Not sure if it's okay to copy other documentation, but if we have to write one from scratch it wouldn't be too different anyway?)

panva · 2024-04-26T09:49:30Z

Wouldn't it be better to expose it under crypto?
Edit: On second thought, maybe not.

I had the exact same thought process.

The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm.

I agree this documentation bit or something to its effect is necessary to circumvent the same brain exercises myself and @lpinca went through :)

joyeecheung · 2024-04-26T09:51:26Z

The doc currently puts it as

The CRC algorithm is designed to compute checksums and to detect error
in data transmission. It's not suitable for cryptographic authentication.

Also, note that both Python and Ruby puts crc32 under a zlib module

github-actions · 2024-04-26T12:20:55Z

Failed to start CI

 ⚠ Something was pushed to the Pull Request branch since the last approving review.
✘ Refusing to run CI on potentially unsafe PR

https://github.com/nodejs/node/actions/runs/8848132665

joyeecheung · 2024-04-26T12:24:33Z

huh, this is...amusing. I'll start from Jenkins.

nodejs-github-bot · 2024-04-26T12:24:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58716/

Co-authored-by: Anna Henningsen <github@addaleax.net>

nodejs-github-bot · 2024-04-26T17:32:22Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58721/

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: #52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 lib,src: * remove --experimental-policy (Rafael Gonzaga) #52583 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: TODO

Notable changes: cli: * (SEMVER-MINOR) allow running wasm in limited vmem with --disable-wasm-trap-handler (Joyee Cheung) #52766 doc: * add pimterry to collaborators (Tim Perry) #52874 fs: * (SEMVER-MINOR) allow 'withFileTypes' to be used with globs (Aviv Keller) #52837 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 lib,src: * remove --experimental-policy (Rafael Gonzaga) #52583 perf_hooks: * (SEMVER-MINOR) add `deliveryType` and `responseStatus` fields (Matthew Aitken) #51589 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #52971

nodejs/node#52692

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: #52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) #52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) #52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) #52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #53486

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: nodejs#52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes: cli: * (SEMVER-MINOR) allow running wasm in limited vmem with --disable-wasm-trap-handler (Joyee Cheung) nodejs#52766 doc: * add pimterry to collaborators (Tim Perry) nodejs#52874 fs: * (SEMVER-MINOR) allow 'withFileTypes' to be used with globs (Aviv Keller) nodejs#52837 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) nodejs#52734 lib,src: * remove --experimental-policy (Rafael Gonzaga) nodejs#52583 perf_hooks: * (SEMVER-MINOR) add `deliveryType` and `responseStatus` fields (Matthew Aitken) nodejs#51589 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) nodejs#52860 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) nodejs#52692 PR-URL: nodejs#52971

Notable changes: doc: * add pimterry to collaborators (Tim Perry) nodejs#52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) nodejs#52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) nodejs#52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) nodejs#52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) nodejs#52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) nodejs#52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) nodejs#52692 PR-URL: nodejs#53486

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib subsystem. labels Apr 25, 2024

anonrig reviewed Apr 25, 2024
View reviewed changes

Comment threadsrc/node_zlib.cc Outdated

anonrig approved these changes Apr 25, 2024
View reviewed changes

joyeecheung force-pushed the crc32 branch 2 times, most recently from b2b9ece to 23223c6Compare April 25, 2024 19:05

joyeecheung force-pushed the crc32 branch from 23223c6 to 53fb0e4Compare April 25, 2024 19:16

joyeecheung commented Apr 25, 2024
View reviewed changes

Comment threaddoc/api/zlib.md

joyeecheung added 2 commits April 25, 2024 22:01

fixup! zlib: expose zlib.crc32()
d9057fc

fixup! fixup! zlib: expose zlib.crc32()
d4fc5ca

joyeecheung force-pushed the crc32 branch from 278dfef to d4fc5caCompare April 25, 2024 20:04

fixup! fixup! zlib: expose zlib.crc32()
b26c0c6

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 26, 2024

github-actionsBot added request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. and removed request-ci Add this label to start a Jenkins CI on a PR. labels Apr 26, 2024

joyeecheung mentioned this pull request Apr 26, 2024
feat(ncu-ci)!: require --certify-safe flag nodejs/node-core-utils#798
Merged

lpinca approved these changes Apr 26, 2024
View reviewed changes

lpinca added the semver-minor PRs that contain new features and should be released in the next minor version. label Apr 26, 2024

addaleax approved these changes Apr 26, 2024
View reviewed changes

Comment threadsrc/node_zlib.cc Outdated

fixup! zlib: expose zlib.crc32()
905c481
Co-authored-by: Anna Henningsen <github@addaleax.net>

targos mentioned this pull request May 13, 2024
v22.2.0 release proposal #52971
Merged

scagood added a commit to eslint-community/eslint-plugin-n that referenced this pull request May 17, 2024
feat(node-builtin): Add node 22.2.0 support
15e090d
nodejs/node#52692

scagood mentioned this pull request May 17, 2024
feat(node-builtin): Add node 22.2.0 support eslint-community/eslint-plugin-n#282
Merged

aladdin-add pushed a commit to eslint-community/eslint-plugin-n that referenced this pull request May 17, 2024
feat(node-builtin): Add node 22.2.0 support (#282)
5221c40
nodejs/node#52692

marco-ippolito mentioned this pull request Jun 17, 2024
v20.15.0 proposal #53486
Merged

segfault-bilibili mentioned this pull request Jul 10, 2024
Incorporate suite of hash utilities into node #2444
Closed

trivikr mentioned this pull request Sep 23, 2024
chore: add Node.js CRC32 for comparison trivikr/benchmark-aws-crt-crc32#2
Merged

trivikr mentioned this pull request Nov 7, 2024
feat(middleware-flexible-checksums): use Node.js native CRC32 checksum API aws/aws-sdk-js-v3#6641
Merged

coderabbitaiBot mentioned this pull request Jul 28, 2026
test(chat-avatars): durable e2e specs for group avatars (workspace#033-chat-avatars) alkem-io/test-suites#597
Open

Navigation Menu

Uh oh!

Uh oh!

zlib: expose zlib.crc32() - #52692

zlib: expose zlib.crc32()#52692
nodejs-github-bot merged 6 commits into
nodejs:mainfrom
joyeecheung:crc32

joyeecheung commented Apr 25, 2024•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

lpinca commented Apr 25, 2024•
edited
Loading

Uh oh!

joyeecheung commented Apr 25, 2024•
edited
Loading

Uh oh!

panva commented Apr 26, 2024

Uh oh!

joyeecheung commented Apr 26, 2024•
edited
Loading

Uh oh!

github-actionsBot commented Apr 26, 2024

Uh oh!

joyeecheung commented Apr 26, 2024

Uh oh!

nodejs-github-bot commented Apr 26, 2024

Uh oh!

Uh oh!

nodejs-github-bot commented Apr 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Uh oh!

Uh oh!

Conversation

joyeecheung commented Apr 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lpinca commented Apr 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joyeecheung commented Apr 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panva commented Apr 26, 2024

Uh oh!

joyeecheung commented Apr 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Apr 26, 2024

Uh oh!

joyeecheung commented Apr 26, 2024

Uh oh!

nodejs-github-bot commented Apr 26, 2024

Uh oh!

Uh oh!

nodejs-github-bot commented Apr 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

joyeecheung commented Apr 25, 2024•
edited
Loading

lpinca commented Apr 25, 2024•
edited
Loading

joyeecheung commented Apr 25, 2024•
edited
Loading

joyeecheung commented Apr 26, 2024•
edited
Loading