Skip to content

zstd: Skip random data faster - #147

Merged
klauspost merged 4 commits into
masterfrom
zstd-skip-random-faster
Aug 14, 2019
Merged

zstd: Skip random data faster#147
klauspost merged 4 commits into
masterfrom
zstd-skip-random-faster

Conversation

@klauspost

Copy link
Copy Markdown
Owner

If we did not get a single match, assume that the data cannot be entropy compressed.

It is very likely that the data is random, and in the rare case it is not, we can live with it.

Before:

BenchmarkRandomEncodeAllFastest-12 100 10475190 ns/op 1001.01 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncodeAllDefault-12 100 11075653 ns/op 946.74 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncoderFastest-12 200 8126950 ns/op 1290.25 MB/s 139 B/op 2 allocs/op
BenchmarkRandomEncoderDefault-12 200 8059989 ns/op 1300.96 MB/s 90 B/op 2 allocs/op

After:

BenchmarkRandomEncodeAllFastest-12 300 4425967 ns/op 2369.15 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncodeAllDefault-12 300 5108791 ns/op 2052.49 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncoderFastest-12 300 5085627 ns/op 2061.84 MB/s 89 B/op 2 allocs/op
BenchmarkRandomEncoderDefault-12 300 5947031 ns/op 1763.19 MB/s 65 B/op 2 allocs/op

If we did not get a single match, assume that the data cannot be entropy compressed.
It is very likely that the data is random, and in the rare case it is not, we can live with it.
Before:
```
BenchmarkRandomEncodeAllFastest-12 100 10475190 ns/op 1001.01 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncodeAllDefault-12 100 11075653 ns/op 946.74 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncoderFastest-12 200 8126950 ns/op 1290.25 MB/s 139 B/op 2 allocs/op
BenchmarkRandomEncoderDefault-12 200 8059989 ns/op 1300.96 MB/s 90 B/op 2 allocs/op
```
After:
```
BenchmarkRandomEncodeAllFastest-12 300 4425967 ns/op 2369.15 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncodeAllDefault-12 300 5108791 ns/op 2052.49 MB/s 0 B/op 0 allocs/op
BenchmarkRandomEncoderFastest-12 300 5085627 ns/op 2061.84 MB/s 89 B/op 2 allocs/op
BenchmarkRandomEncoderDefault-12 300 5947031 ns/op 1763.19 MB/s 65 B/op 2 allocs/op
```
@klauspost
klauspost merged commit f82c96c into masterAug 14, 2019
@klauspost
klauspost deleted the zstd-skip-random-faster branch August 14, 2019 10:11
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@klauspost