Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Implement ZStandard Stream, Encoder, Decoder - #119575

Merged
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd
Jan 22, 2026
Merged

Implement ZStandard Stream, Encoder, Decoder#119575
rzikm merged 130 commits into
dotnet:mainfrom
rzikm:59591-zstd

Conversation

@rzikm

@rzikmrzikm commented Sep 11, 2025

Copy link
Copy Markdown
Member

Implements #59591

There are some patches applied to vendored zstd sources which are listed in src/native/zstd-version.txt, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results
MethodlevelfileAlgorithmMeanErrorStdDevSizeCompressed
CompressOptimalalice29.txtBrotli2,306.40 μs45.807 μs71.316 μs14848154517
CompressOptimalalice29.txtZstandard842.743 μs8.3719 μs7.8310 μs14848154919
CompressOptimalalice29.txtDeflate2,421.16 μs41.274 μs38.608 μs14848153890
DecompressOptimalalice29.txtBrotli320.82 μs4.802 μs4.257 μs
DecompressOptimalalice29.txtZstandard139.251 μs2.0885 μs1.7440 μs
DecompressOptimalalice29.txtDeflate275.43 μs4.849 μs4.535 μs
CompressOptimalsumBrotli577.62 μs7.647 μs5.971 μs3824012527
CompressOptimalsumZstandard155.050 μs3.0367 μs3.3753 μs3824013389
CompressOptimalsumDeflate404.70 μs7.732 μs9.779 μs3824013388
DecompressOptimalsumBrotli101.26 μs2.781 μs7.566 μs
DecompressOptimalsumZstandard31.333 μs0.6172 μs0.5774 μs
DecompressOptimalsumDeflate54.86 μs0.676 μs0.632 μs
CompressOptimalTestDocument.pdfBrotli763.43 μs13.851 μs23.520 μs121993115863
CompressOptimalTestDocument.pdfZstandard182.115 μs3.3456 μs2.9658 μs121993119285
CompressOptimalTestDocument.pdfDeflate2,133.16 μs42.543 μs104.358 μs121993115369
DecompressOptimalTestDocument.pdfBrotli267.64 μs5.281 μs10.047 μs
DecompressOptimalTestDocument.pdfZstandard17.508 μs0.3336 μs0.3708 μs
DecompressOptimalTestDocument.pdfDeflate224.92 μs2.642 μs2.342 μs
CompressFastestalice29.txtBrotli732.30 μs14.288 μs19.074 μs14848159265
CompressFastestalice29.txtZstandard509.3 us5.10 us4.53 us14848159891
CompressFastestalice29.txtDeflate612.28 μs11.300 μs16.564 μs14848183053
DecompressFastestalice29.txtBrotli380.92 μs7.570 μs9.844 μs
DecompressFastestalice29.txtZstandard111.14 us1.321 us1.103 us
DecompressFastestalice29.txtDeflate361.78 μs6.700 μs14.847 μs
CompressFastestsumBrotli125.26 μs2.489 μs3.801 μs3824014305
CompressFastestsumZstandard108.0 us1.05 us0.88 us3824013848
CompressFastestsumDeflate121.81 μs2.409 μs4.699 μs3824017222
DecompressFastestsumBrotli99.19 μs1.974 μs3.243 μs
DecompressFastestsumZstandard29.01 us0.559 us0.599 us
DecompressFastestsumDeflate58.68 μs0.999 μs0.934 μs
CompressFastestTestDocument.pdfBrotli300.40 μs5.753 μs6.394 μs121993117104
CompressFastestTestDocument.pdfZstandard137.3 us2.00 us1.67 us121993119405
CompressFastestTestDocument.pdfDeflate947.61 μs17.861 μs18.342 μs121993121404
DecompressFastestTestDocument.pdfBrotli233.14 μs3.625 μs3.391 μs
DecompressFastestTestDocument.pdfZstandard17.22 us0.260 us0.244 us
DecompressFastestTestDocument.pdfDeflate238.81 μs2.153 μs1.908 μs

Supported platforms (verified by CI): Windows, Linux, OSX, Mobile

  • WASM support should be possible, but will need a bit of work to make it opt-in to avoid increasing the runtime size. Will be left for follow-up.

Binary size diff (local debug build, Windows, main vs PR):

NameBeforeAfterDiff
System.IO.Compression.Native.dll4 861 9526 029 8241 167 872
System.IO.Compression.Zstandard.dll054 78454 784

we can probably shave off about 130 kB by removing legacy support (support for formats used by previous zstd versions, see https://deepwiki.com/facebook/zstd/15-legacy-support), but I am not sure if that is worth it.

I wasnt able to check trimmed size of a single-file application, but I expect that roughly +1.1 MB increase due to lack of native pruning capabilities.

Comment threadsrc/libraries/System.IO.Compression.ZStandard/README.md Outdated
Comment threadsrc/native/external/zstd-version.txt Outdated
Comment threadsrc/native/external/zstd.patch Outdated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements ZStandard stream, encoder, and decoder functionality for .NET by integrating the zstd native library. The changes primarily consist of adding the external zstd library source files and build infrastructure to support compression and decompression capabilities.

Key Changes:

  • Integration of zstd native library source code (version metadata suggests 0.9.0)
  • Build system configuration files for Meson and CMake
  • Single-file library generation scripts and examples
  • Test infrastructure for the zstd implementation

Reviewed Changes

Copilot reviewed 142 out of 237 changed files in this pull request and generated no comments.

Show a summary per file
FileDescription
src/native/external/zstd/lib/common/fse_decompress.cFSE (Finite State Entropy) decompression implementation
src/native/external/zstd/lib/common/fse.hFSE codec public API and data structures
src/native/external/zstd/lib/common/error_private.hInternal error handling macros and definitions
src/native/external/zstd/lib/common/error_private.cError code to string mapping implementation
src/native/external/zstd/lib/common/entropy_common.cCommon entropy encoding/decoding functions
src/native/external/zstd/lib/common/debug.hDebug logging and assertion macros
src/native/external/zstd/lib/common/debug.cDebug level global variable definition
src/native/external/zstd/lib/common/cpu.hCPU feature detection (BMI2, AVX, etc.)
src/native/external/zstd/lib/common/compiler.hCompiler-specific macros and attributes
src/native/external/zstd/lib/common/bitstream.hBitstream encoding/decoding utilities
src/native/external/zstd/lib/common/bits.hBit manipulation helper functions
src/native/external/zstd/lib/common/allocations.hCustom memory allocation wrappers
src/native/external/zstd/lib/README.mdDocumentation for library structure and build options
src/native/external/zstd/lib/MakefileBuild configuration for the zstd library
src/native/external/zstd/lib/BUCKBuck build system configuration
src/native/external/zstd/lib/.gitignoreGit ignore rules for build artifacts
src/native/external/zstd/build/single_file_libs/*Scripts and examples for single-file library generation
src/native/external/zstd/build/meson/*Meson build system configuration files
src/native/external/zstd/build/cmake/tests/.gitignoreGit ignore rules for CMake test artifacts

@rzikm

Copy link
Copy Markdown
MemberAuthor

I think this PR is ready for the first round of reviews

@rzikm
rzikm requested review from a team and stephentoubNovember 13, 2025 16:51
@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/ba-g Test failures are unrelated

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated no new comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@rzikm

Copy link
Copy Markdown
MemberAuthor

@gewarren Could you please take a look at the documentation comments if it is consistent with what we generally write?

CC also @blowdart@GrabYourPitchforks who also wanted to look at the comments.

@rzikm

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@gewarrengewarren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzikm I left some suggestions, thanks for tagging me.

Comment threadsrc/libraries/System.IO.Compression.Zstandard/src/Resources/Strings.resx Outdated
This reverts commit c0917ab.
Revert "fixup! Enable on WASI/Browser"
This reverts commit 2a5e1db.
Revert "Enable on WASI/Browser"
This reverts commit 6013b24.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

12 participants

@rzikm@jkotas@pavelsavara@vitek-karas@stephentoub@am11@ericstj@bartonjs@gewarren@MihaZupan@karelz