Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454

Merged
carlossanlop merged 6 commits into
dotnet:mainfrom
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged

Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom
carlossanlop:zlib-ng-reattempt

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Jul 4, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Contributes to: #101465

This PR reapplies the reverted changes due to a wasm break in the official build: #102403

It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.

@carlossanlopcarlossanlop self-assigned this Jul 4, 2024
@dotnet-policy-service

dotnet-policy-serviceBot commented Jul 4, 2024

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

jkotas
jkotas reviewed Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake Outdated
carlossanlop
carlossanlop commented Jul 4, 2024
View reviewed changes

@carlossanlopcarlossanlop left a comment

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.

I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:

docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:

 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake Outdated
Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt Outdated
Comment threadsrc/native/external/zlib-ng.cmake Outdated
Comment threadsrc/native/external/zlib-ng.cmake Outdated
jkotas
jkotas reviewed Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake Outdated
This was referenced Jul 5, 2024
Closed
Closed
Closed
Closed
@carlossanlop

carlossanlop commented Jul 5, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:

https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114

@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines

azure-pipelinesBot commented Jul 8, 2024

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024
Open
Closed
Open
@pavelsavara

pavelsavara commented Jul 8, 2024•
edited
Loading

Copy link
Copy Markdown
Member

zlib-ng is not ready for WASI Preview2, see #104414 (comment)

@ericstj

ericstj commented Jul 8, 2024

Copy link
Copy Markdown
Member

zlib-ng is not ready for WASI Preview2, see #104414 (comment)

Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.

ericstj
ericstj approved these changes Jul 8, 2024
View reviewed changes

@ericstjericstj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.

Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
<PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
<!-- zlib-specific files -->
<PlatformManifestFileEntry Include="libz.a" IsNative="true" />
<PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />

@ericstjericstjJul 8, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.

I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249

I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.

@carlossanlopcarlossanlopJul 8, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.

@jkotasjkotasJul 13, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?

zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.

@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.

@carlossanlop
carlossanlop enabled auto-merge (squash) July 8, 2024 23:04
@carlossanlopcarlossanlop mentioned this pull request Jul 8, 2024
Closed
@carlossanlop

carlossanlop commented Jul 8, 2024

Copy link
Copy Markdown
ContributorAuthor

/ba-g All failures determined to be unrelated.

@carlossanlop
carlossanlop merged commit cf08d43 into dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop deleted the zlib-ng-reattempt branch July 8, 2024 23:11
@carlossanlop

carlossanlop commented Jul 9, 2024

Copy link
Copy Markdown
ContributorAuthor

@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?

@LoopedBard3

LoopedBard3 commented Jul 10, 2024

Copy link
Copy Markdown
Member

If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.

@am11

am11 commented Jul 10, 2024

Copy link
Copy Markdown
Member

@carlossanlop, VMR jobs are failing with:

 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?

@carlossanlop

carlossanlop commented Jul 10, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Yep, I've been notified and I'm looking into this.

Do we need to set -p:UseSystemZlib=true?

No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.

@LoopedBard3

LoopedBard3 commented Jul 11, 2024

Copy link
Copy Markdown
Member

Likely related improvements:
Linux arm64: dotnet/perf-autofiling-issues#38054

@am11am11 mentioned this pull request Jul 12, 2024
Merged
This was referenced Jul 16, 2024
Closed
Closed
Closed
@LoopedBard3LoopedBard3 mentioned this pull request Jul 16, 2024
Open
This was referenced Jul 16, 2024
Closed
Closed
@LoopedBard3LoopedBard3 mentioned this pull request Jul 18, 2024
Closed
@sebastienros

sebastienros commented Jul 19, 2024

Copy link
Copy Markdown
Member

We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.

@stephentoub

stephentoub commented Jul 19, 2024

Copy link
Copy Markdown
Member

We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.

There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.

@jkurdekjkurdek mentioned this pull request Aug 7, 2024
Closed
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454
Merged
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged
Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Conversation
@carlossanlop
@carlossanlopcarlossanlop
commented
Jul 4, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Contributes to: #101465
This PR reapplies the reverted changes due to a wasm break in the official build: #102403
It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.
@carlossanlopcarlossanlop
self-assigned this
Jul 4, 2024
@dotnet-policy-service
dotnet-policy-serviceBot
commented
Jul 4, 2024

Copy link


Copy Markdown

Contributor
Tagging subscribers to this area: @dotnet/area-system-io-compression

See info in area-owners.md if you want to be subscribed.
jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake
Outdated

carlossanlop
carlossanlop
commented
Jul 4, 2024
View reviewed changes
@carlossanlopcarlossanlop
left a comment


Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.
I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:
docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:
 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake
Outdated

Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake
Outdated


This was referenced Jul 5, 2024

Closed


Closed


Closed


Closed

@carlossanlop
carlossanlop
commented
Jul 5, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:
https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114
@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines
azure-pipelinesBot
commented
Jul 8, 2024

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024

Open


Closed


Open

@pavelsavara
pavelsavara
commented
Jul 8, 2024•

edited

Loading

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
@ericstj
ericstj
commented
Jul 8, 2024

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.
ericstj
ericstj
approved these changes
Jul 8, 2024
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.
Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
 <PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
 <!-- zlib-specific files -->
 <PlatformManifestFileEntry Include="libz.a" IsNative="true" />
 <PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />
@ericstjericstjJul 8, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.
I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249
I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.
@carlossanlopcarlossanlopJul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.
@jkotasjkotasJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?
zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.
@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.
@carlossanlop
carlossanlop
enabled auto-merge (squash)
July 8, 2024 23:04
@carlossanlopcarlossanlop
mentioned this pull request
Jul 8, 2024

Closed

@carlossanlop
carlossanlop
commented
Jul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
/ba-g All failures determined to be unrelated.
@carlossanlop
carlossanlop
merged commit cf08d43
into
dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop
deleted the

zlib-ng-reattempt

branch
July 8, 2024 23:11
@carlossanlop
carlossanlop
commented
Jul 9, 2024

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?
@LoopedBard3
LoopedBard3
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.
@am11
am11
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
@carlossanlop, VMR jobs are failing with:
 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?
@carlossanlop
carlossanlop
commented
Jul 10, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yep, I've been notified and I'm looking into this.
Do we need to set -p:UseSystemZlib=true?
No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.
@LoopedBard3
LoopedBard3
commented
Jul 11, 2024

Copy link


Copy Markdown

Member
Likely related improvements:

Linux arm64: dotnet/perf-autofiling-issues#38054
@am11am11
mentioned this pull request
Jul 12, 2024
 Merged


This was referenced Jul 16, 2024

Closed


Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 16, 2024

Open


This was referenced Jul 16, 2024

Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 18, 2024

Closed

@sebastienros
sebastienros
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
@stephentoub
stephentoub
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.
@jkurdekjkurdek
mentioned this pull request
Aug 7, 2024

Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454
Merged
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged
Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Conversation
@carlossanlop
@carlossanlopcarlossanlop
commented
Jul 4, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Contributes to: #101465
This PR reapplies the reverted changes due to a wasm break in the official build: #102403
It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.
@carlossanlopcarlossanlop
self-assigned this
Jul 4, 2024
@dotnet-policy-service
dotnet-policy-serviceBot
commented
Jul 4, 2024

Copy link


Copy Markdown

Contributor
Tagging subscribers to this area: @dotnet/area-system-io-compression

See info in area-owners.md if you want to be subscribed.
jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake
Outdated

carlossanlop
carlossanlop
commented
Jul 4, 2024
View reviewed changes
@carlossanlopcarlossanlop
left a comment


Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.
I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:
docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:
 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake
Outdated

Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake
Outdated


This was referenced Jul 5, 2024

Closed


Closed


Closed


Closed

@carlossanlop
carlossanlop
commented
Jul 5, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:
https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114
@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines
azure-pipelinesBot
commented
Jul 8, 2024

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024

Open


Closed


Open

@pavelsavara
pavelsavara
commented
Jul 8, 2024•

edited

Loading

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
@ericstj
ericstj
commented
Jul 8, 2024

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.
ericstj
ericstj
approved these changes
Jul 8, 2024
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.
Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
 <PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
 <!-- zlib-specific files -->
 <PlatformManifestFileEntry Include="libz.a" IsNative="true" />
 <PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />
@ericstjericstjJul 8, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.
I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249
I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.
@carlossanlopcarlossanlopJul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.
@jkotasjkotasJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?
zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.
@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.
@carlossanlop
carlossanlop
enabled auto-merge (squash)
July 8, 2024 23:04
@carlossanlopcarlossanlop
mentioned this pull request
Jul 8, 2024

Closed

@carlossanlop
carlossanlop
commented
Jul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
/ba-g All failures determined to be unrelated.
@carlossanlop
carlossanlop
merged commit cf08d43
into
dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop
deleted the

zlib-ng-reattempt

branch
July 8, 2024 23:11
@carlossanlop
carlossanlop
commented
Jul 9, 2024

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?
@LoopedBard3
LoopedBard3
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.
@am11
am11
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
@carlossanlop, VMR jobs are failing with:
 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?
@carlossanlop
carlossanlop
commented
Jul 10, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yep, I've been notified and I'm looking into this.
Do we need to set -p:UseSystemZlib=true?
No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.
@LoopedBard3
LoopedBard3
commented
Jul 11, 2024

Copy link


Copy Markdown

Member
Likely related improvements:

Linux arm64: dotnet/perf-autofiling-issues#38054
@am11am11
mentioned this pull request
Jul 12, 2024
 Merged


This was referenced Jul 16, 2024

Closed


Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 16, 2024

Open


This was referenced Jul 16, 2024

Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 18, 2024

Closed

@sebastienros
sebastienros
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
@stephentoub
stephentoub
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.
@jkurdekjkurdek
mentioned this pull request
Aug 7, 2024

Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454
Merged
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged
Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Conversation
@carlossanlop
@carlossanlopcarlossanlop
commented
Jul 4, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Contributes to: #101465
This PR reapplies the reverted changes due to a wasm break in the official build: #102403
It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.
@carlossanlopcarlossanlop
self-assigned this
Jul 4, 2024
@dotnet-policy-service
dotnet-policy-serviceBot
commented
Jul 4, 2024

Copy link


Copy Markdown

Contributor
Tagging subscribers to this area: @dotnet/area-system-io-compression

See info in area-owners.md if you want to be subscribed.
jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake
Outdated

carlossanlop
carlossanlop
commented
Jul 4, 2024
View reviewed changes
@carlossanlopcarlossanlop
left a comment


Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.
I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:
docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:
 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake
Outdated

Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake
Outdated


This was referenced Jul 5, 2024

Closed


Closed


Closed


Closed

@carlossanlop
carlossanlop
commented
Jul 5, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:
https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114
@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines
azure-pipelinesBot
commented
Jul 8, 2024

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024

Open


Closed


Open

@pavelsavara
pavelsavara
commented
Jul 8, 2024•

edited

Loading

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
@ericstj
ericstj
commented
Jul 8, 2024

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.
ericstj
ericstj
approved these changes
Jul 8, 2024
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.
Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
 <PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
 <!-- zlib-specific files -->
 <PlatformManifestFileEntry Include="libz.a" IsNative="true" />
 <PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />
@ericstjericstjJul 8, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.
I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249
I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.
@carlossanlopcarlossanlopJul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.
@jkotasjkotasJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?
zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.
@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.
@carlossanlop
carlossanlop
enabled auto-merge (squash)
July 8, 2024 23:04
@carlossanlopcarlossanlop
mentioned this pull request
Jul 8, 2024

Closed

@carlossanlop
carlossanlop
commented
Jul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
/ba-g All failures determined to be unrelated.
@carlossanlop
carlossanlop
merged commit cf08d43
into
dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop
deleted the

zlib-ng-reattempt

branch
July 8, 2024 23:11
@carlossanlop
carlossanlop
commented
Jul 9, 2024

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?
@LoopedBard3
LoopedBard3
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.
@am11
am11
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
@carlossanlop, VMR jobs are failing with:
 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?
@carlossanlop
carlossanlop
commented
Jul 10, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yep, I've been notified and I'm looking into this.
Do we need to set -p:UseSystemZlib=true?
No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.
@LoopedBard3
LoopedBard3
commented
Jul 11, 2024

Copy link


Copy Markdown

Member
Likely related improvements:

Linux arm64: dotnet/perf-autofiling-issues#38054
@am11am11
mentioned this pull request
Jul 12, 2024
 Merged


This was referenced Jul 16, 2024

Closed


Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 16, 2024

Open


This was referenced Jul 16, 2024

Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 18, 2024

Closed

@sebastienros
sebastienros
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
@stephentoub
stephentoub
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.
@jkurdekjkurdek
mentioned this pull request
Aug 7, 2024

Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454
Merged
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged
Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Conversation
@carlossanlop
@carlossanlopcarlossanlop
commented
Jul 4, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Contributes to: #101465
This PR reapplies the reverted changes due to a wasm break in the official build: #102403
It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.
@carlossanlopcarlossanlop
self-assigned this
Jul 4, 2024
@dotnet-policy-service
dotnet-policy-serviceBot
commented
Jul 4, 2024

Copy link


Copy Markdown

Contributor
Tagging subscribers to this area: @dotnet/area-system-io-compression

See info in area-owners.md if you want to be subscribed.
jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake
Outdated

carlossanlop
carlossanlop
commented
Jul 4, 2024
View reviewed changes
@carlossanlopcarlossanlop
left a comment


Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.
I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:
docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:
 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake
Outdated

Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake
Outdated


This was referenced Jul 5, 2024

Closed


Closed


Closed


Closed

@carlossanlop
carlossanlop
commented
Jul 5, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:
https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114
@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines
azure-pipelinesBot
commented
Jul 8, 2024

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024

Open


Closed


Open

@pavelsavara
pavelsavara
commented
Jul 8, 2024•

edited

Loading

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
@ericstj
ericstj
commented
Jul 8, 2024

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.
ericstj
ericstj
approved these changes
Jul 8, 2024
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.
Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
 <PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
 <!-- zlib-specific files -->
 <PlatformManifestFileEntry Include="libz.a" IsNative="true" />
 <PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />
@ericstjericstjJul 8, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.
I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249
I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.
@carlossanlopcarlossanlopJul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.
@jkotasjkotasJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?
zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.
@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.
@carlossanlop
carlossanlop
enabled auto-merge (squash)
July 8, 2024 23:04
@carlossanlopcarlossanlop
mentioned this pull request
Jul 8, 2024

Closed

@carlossanlop
carlossanlop
commented
Jul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
/ba-g All failures determined to be unrelated.
@carlossanlop
carlossanlop
merged commit cf08d43
into
dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop
deleted the

zlib-ng-reattempt

branch
July 8, 2024 23:11
@carlossanlop
carlossanlop
commented
Jul 9, 2024

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?
@LoopedBard3
LoopedBard3
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.
@am11
am11
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
@carlossanlop, VMR jobs are failing with:
 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?
@carlossanlop
carlossanlop
commented
Jul 10, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yep, I've been notified and I'm looking into this.
Do we need to set -p:UseSystemZlib=true?
No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.
@LoopedBard3
LoopedBard3
commented
Jul 11, 2024

Copy link


Copy Markdown

Member
Likely related improvements:

Linux arm64: dotnet/perf-autofiling-issues#38054
@am11am11
mentioned this pull request
Jul 12, 2024
 Merged


This was referenced Jul 16, 2024

Closed


Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 16, 2024

Open


This was referenced Jul 16, 2024

Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 18, 2024

Closed

@sebastienros
sebastienros
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
@stephentoub
stephentoub
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.
@jkurdekjkurdek
mentioned this pull request
Aug 7, 2024

Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454
Merged
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged
Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Conversation
@carlossanlop
@carlossanlopcarlossanlop
commented
Jul 4, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Contributes to: #101465
This PR reapplies the reverted changes due to a wasm break in the official build: #102403
It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.
@carlossanlopcarlossanlop
self-assigned this
Jul 4, 2024
@dotnet-policy-service
dotnet-policy-serviceBot
commented
Jul 4, 2024

Copy link


Copy Markdown

Contributor
Tagging subscribers to this area: @dotnet/area-system-io-compression

See info in area-owners.md if you want to be subscribed.
jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake
Outdated

carlossanlop
carlossanlop
commented
Jul 4, 2024
View reviewed changes
@carlossanlopcarlossanlop
left a comment


Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.
I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:
docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:
 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake
Outdated

Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake
Outdated


This was referenced Jul 5, 2024

Closed


Closed


Closed


Closed

@carlossanlop
carlossanlop
commented
Jul 5, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:
https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114
@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines
azure-pipelinesBot
commented
Jul 8, 2024

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024

Open


Closed


Open

@pavelsavara
pavelsavara
commented
Jul 8, 2024•

edited

Loading

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
@ericstj
ericstj
commented
Jul 8, 2024

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.
ericstj
ericstj
approved these changes
Jul 8, 2024
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.
Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
 <PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
 <!-- zlib-specific files -->
 <PlatformManifestFileEntry Include="libz.a" IsNative="true" />
 <PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />
@ericstjericstjJul 8, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.
I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249
I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.
@carlossanlopcarlossanlopJul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.
@jkotasjkotasJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?
zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.
@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.
@carlossanlop
carlossanlop
enabled auto-merge (squash)
July 8, 2024 23:04
@carlossanlopcarlossanlop
mentioned this pull request
Jul 8, 2024

Closed

@carlossanlop
carlossanlop
commented
Jul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
/ba-g All failures determined to be unrelated.
@carlossanlop
carlossanlop
merged commit cf08d43
into
dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop
deleted the

zlib-ng-reattempt

branch
July 8, 2024 23:11
@carlossanlop
carlossanlop
commented
Jul 9, 2024

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?
@LoopedBard3
LoopedBard3
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.
@am11
am11
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
@carlossanlop, VMR jobs are failing with:
 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?
@carlossanlop
carlossanlop
commented
Jul 10, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yep, I've been notified and I'm looking into this.
Do we need to set -p:UseSystemZlib=true?
No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.
@LoopedBard3
LoopedBard3
commented
Jul 11, 2024

Copy link


Copy Markdown

Member
Likely related improvements:

Linux arm64: dotnet/perf-autofiling-issues#38054
@am11am11
mentioned this pull request
Jul 12, 2024
 Merged


This was referenced Jul 16, 2024

Closed


Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 16, 2024

Open


This was referenced Jul 16, 2024

Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 18, 2024

Closed

@sebastienros
sebastienros
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
@stephentoub
stephentoub
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.
@jkurdekjkurdek
mentioned this pull request
Aug 7, 2024

Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454
Merged
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged
Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Conversation
@carlossanlop
@carlossanlopcarlossanlop
commented
Jul 4, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Contributes to: #101465
This PR reapplies the reverted changes due to a wasm break in the official build: #102403
It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.
@carlossanlopcarlossanlop
self-assigned this
Jul 4, 2024
@dotnet-policy-service
dotnet-policy-serviceBot
commented
Jul 4, 2024

Copy link


Copy Markdown

Contributor
Tagging subscribers to this area: @dotnet/area-system-io-compression

See info in area-owners.md if you want to be subscribed.
jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake
Outdated

carlossanlop
carlossanlop
commented
Jul 4, 2024
View reviewed changes
@carlossanlopcarlossanlop
left a comment


Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.
I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:
docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:
 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake
Outdated

Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake
Outdated


This was referenced Jul 5, 2024

Closed


Closed


Closed


Closed

@carlossanlop
carlossanlop
commented
Jul 5, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:
https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114
@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines
azure-pipelinesBot
commented
Jul 8, 2024

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024

Open


Closed


Open

@pavelsavara
pavelsavara
commented
Jul 8, 2024•

edited

Loading

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
@ericstj
ericstj
commented
Jul 8, 2024

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.
ericstj
ericstj
approved these changes
Jul 8, 2024
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.
Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
 <PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
 <!-- zlib-specific files -->
 <PlatformManifestFileEntry Include="libz.a" IsNative="true" />
 <PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />
@ericstjericstjJul 8, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.
I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249
I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.
@carlossanlopcarlossanlopJul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.
@jkotasjkotasJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?
zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.
@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.
@carlossanlop
carlossanlop
enabled auto-merge (squash)
July 8, 2024 23:04
@carlossanlopcarlossanlop
mentioned this pull request
Jul 8, 2024

Closed

@carlossanlop
carlossanlop
commented
Jul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
/ba-g All failures determined to be unrelated.
@carlossanlop
carlossanlop
merged commit cf08d43
into
dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop
deleted the

zlib-ng-reattempt

branch
July 8, 2024 23:11
@carlossanlop
carlossanlop
commented
Jul 9, 2024

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?
@LoopedBard3
LoopedBard3
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.
@am11
am11
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
@carlossanlop, VMR jobs are failing with:
 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?
@carlossanlop
carlossanlop
commented
Jul 10, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yep, I've been notified and I'm looking into this.
Do we need to set -p:UseSystemZlib=true?
No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.
@LoopedBard3
LoopedBard3
commented
Jul 11, 2024

Copy link


Copy Markdown

Member
Likely related improvements:

Linux arm64: dotnet/perf-autofiling-issues#38054
@am11am11
mentioned this pull request
Jul 12, 2024
 Merged


This was referenced Jul 16, 2024

Closed


Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 16, 2024

Open


This was referenced Jul 16, 2024

Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 18, 2024

Closed

@sebastienros
sebastienros
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
@stephentoub
stephentoub
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.
@jkurdekjkurdek
mentioned this pull request
Aug 7, 2024

Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Migrate to zlib-ng, part 2: consume it in runtime (second attempt) - #104454
Merged
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Jul 8, 2024
Merged
Migrate to zlib-ng, part 2: consume it in runtime (second attempt)#104454
carlossanlop merged 6 commits into
dotnet:mainfrom 
carlossanlop:zlib-ng-reattempt
Conversation
@carlossanlop
@carlossanlopcarlossanlop
commented
Jul 4, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Contributes to: #101465
This PR reapplies the reverted changes due to a wasm break in the official build: #102403
It also includes an extra commit that fixes the NativeAOT build failure that only happened in the official build.
@carlossanlopcarlossanlop
self-assigned this
Jul 4, 2024
@dotnet-policy-service
dotnet-policy-serviceBot
commented
Jul 4, 2024

Copy link


Copy Markdown

Contributor
Tagging subscribers to this area: @dotnet/area-system-io-compression

See info in area-owners.md if you want to be subscribed.
jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadsrc/native/external/zlib-ng.cmake
Outdated

carlossanlop
carlossanlop
commented
Jul 4, 2024
View reviewed changes
@carlossanlopcarlossanlop
left a comment


Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavelsavara@ilonatommy@kg@jkotas@jkoritzinsky@akoeplinger I need some help, I am still unable to get a successful build in wasm.
I can repro the exact same scenario used in the CI by executing these commands inside my Ubuntu WSL:
docker run -ti -d -v /home/carlos/repos/runtime:/home/carlos/repos/runtime -e ROOTFS_DIR=/crossrootfs/x64 -w /home/carlos/repos/runtime mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-webassembly-amd64-net9.0
docker attach <GUID_HERE>
git config --global --add safe.directory /home/carlos/repos/runtime
./build.sh mono+libs+host+packs+libs.tests -c Release -arch wasm -os browser /p:MonoEnableAssertMessages=true /p:BrowserHost=linux /p:AotHostArchitecture=x64 /p:AotHostOS=linux

And the build is failing with the following error, which tells me that I am still unable to successfully link the system zlib (which I confirmed exists inside the crossrootfs folder) to mono/metadata:
 /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:33:10: fatal error: 'zlib.h' file not found
#include <zlib.h>
^~~~~~~~
1 error generated.
emcc: error: '/home/carlos/repos/runtime/src/mono/browser/emsdk/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration --sysroot=/home/carlos/repos/runtime/src/mono/browser/emsdk/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DCOMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DHAVE_SGEN_GC -DMONO_DLL_EXPORT -D_THREAD_SAFE -I/home/carlos/repos/runtime/artifacts/obj -I/home/carlos/repos/runtime/src/native -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/../.. -I/home/carlos/repos/runtime/src/mono/mono/metadata/.. -I/home/carlos/repos/runtime/src/native/public/. -I/home/carlos/repos/runtime/artifacts/obj/mono/browser.wasm.Release/mono/eglib -I/home/carlos/repos/runtime/src/mono/mono/eglib -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -O3 -DNDEBUG -std=gnu11 -g3 -fPIC -fvisibility=hidden -Wno-strict-prototypes -Wno-unused-but-set-variable -Wno-single-bit-bitfield-constant-conversion -Os -ffp-contract=off -MD -MT mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o -MF CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o.d -c /home/carlos/repos/runtime/src/mono/mono/metadata/debug-mono-ppdb.c -o CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o' failed (returned 1)
make[2]: *** [mono/metadata/CMakeFiles/metadata_objects.dir/build.make:257: mono/metadata/CMakeFiles/metadata_objects.dir/debug-mono-ppdb.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:415: mono/metadata/CMakeFiles/metadata_objects.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
/home/carlos/repos/runtime/src/mono/mono.proj(765,5): error MSB3073: The command "cmake --build . --target install --config Release --parallel 12" exited with code 2.
Build FAILED.

Comment threadsrc/native/libs/System.IO.Compression.Native/extra_libs.cmake
Outdated

Comment threadsrc/mono/CMakeLists.txt
Comment threadsrc/mono/mono/metadata/CMakeLists.txt
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

Comment threadsrc/native/external/zlib-ng.cmake
Outdated

jkotas
jkotas
reviewed
Jul 4, 2024
View reviewed changes
Comment threadeng/native/configureplatform.cmake
Outdated


This was referenced Jul 5, 2024

Closed


Closed


Closed


Closed

@carlossanlop
carlossanlop
commented
Jul 5, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3 I think I need to remove zlib from the list of dependencies that get installed by this performance job. Can you please confirm? If yes, what azp run do I need to execute to confirm that change?:
https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/run-performance-job.yml#L114
@carlossanlop
Fix windows nativeaot failure happening when executing:
d232aaf
build.cmd -ci -arch x64 -os windows -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true
Failure message this commit fixes:
[.packages\microsoft.dotnet.sharedframework.sdk\9.0.0-beta.24327.1\targets\sharedfx.targets(296,5):
error : The following files are missing entries in the templated manifest:
error : zlibstatic.lib. Add these file names with extensions to the 'PlatformManifestFileEntry' item group for the runtime pack and corresponding ref pack to include them in the platform manifest.
[\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj]
@azure-pipelines
azure-pipelinesBot
commented
Jul 8, 2024

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Jul 8, 2024

Open


Closed


Open

@pavelsavara
pavelsavara
commented
Jul 8, 2024•

edited

Loading

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
@ericstj
ericstj
commented
Jul 8, 2024

Copy link


Copy Markdown

Member
zlib-ng is not ready for WASI Preview2, see #104414 (comment)
Let's collaborate on getting those fixed in the WASI PR. It will be easier to test and iterate there. It's important that this change makes it into Preview 7.
ericstj
ericstj
approved these changes
Jul 8, 2024
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve this change to get this in and building. Please file a follow up issue on the platform manifest cleanup, and work with @pavelsavara on wasi fix.
Comment threadsrc/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
 <PlatformManifestFileEntry Include="libicuuc.a" IsNative="true" />
 <!-- zlib-specific files -->
 <PlatformManifestFileEntry Include="libz.a" IsNative="true" />
 <PlatformManifestFileEntry Include="zlibstatic.lib" IsNative="true" />
@ericstjericstjJul 8, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework? That seems unusual as it's a dev-time resource - however I see other dev-time resources here like main.c, driver.h, etc. I take it you needed to add this to build - but I bet a better fix is to get all of this stuff out of the platform manifest since I don't think it needs to be fed to the SDK for conflict resolution @jkoritzinsky@dsplaisted.
I see we have a metadata flag ExcludeFromDataFiles for that: https://github.com/dotnet/arcade/blob/76f733ee57811c38bb5b8e1ac9c6c50e92bc5dc9/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets#L249
I think we should review all the things added here, remove files that don't need to ship - if any, and apply ExcludeFromDataFiles for the files that need to ship and don't actually need to go into the manifest. That can be done separately.
@carlossanlopcarlossanlopJul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind as a follow up.
@jkotasjkotasJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it intentional to ship zlibstatic.lib in our shared framework?
zlibstatic.lib ships in the native aot runtime package only. It does not ship in the regular shared framework.
@jkoritzinskyjkoritzinskyJul 13, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the shared framework SDK, no one really knew what needed to go into the manifest, so it requires/lists all files. If we know what types of files need to be in it, I can update the SDK to only validate (for the template case) or include (for the generative case) files that matter.
@carlossanlop
carlossanlop
enabled auto-merge (squash)
July 8, 2024 23:04
@carlossanlopcarlossanlop
mentioned this pull request
Jul 8, 2024

Closed

@carlossanlop
carlossanlop
commented
Jul 8, 2024

Copy link


Copy Markdown

ContributorAuthor
/ba-g All failures determined to be unrelated.
@carlossanlop
carlossanlop
merged commit cf08d43
into
dotnet:mainJul 8, 2024
@carlossanlop
carlossanlop
deleted the

zlib-ng-reattempt

branch
July 8, 2024 23:11
@carlossanlop
carlossanlop
commented
Jul 9, 2024

Copy link


Copy Markdown

ContributorAuthor
@LoopedBard3@caaavik-msft@DrewScoggins where can I see the runtime perf chart before and after this change?
@LoopedBard3
LoopedBard3
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
If you have specific tests in mind, we have an ADX dashboard and database. I have DMed these to you. Otherwise, if you want to just look at a few of the test results, we also have an allTestHistory that is updated once a day with the latest run data. Although it seems we currently have a pathing bug so you will need to remove the "reports/" from the url to view the actual test if the links give a 404.
@am11
am11
commented
Jul 10, 2024

Copy link


Copy Markdown

Member
@carlossanlop, VMR jobs are failing with:
 Generating native code
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:122:(.text.CompressionNative_DeflateInit2_+0x89): undefined reference to `deflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Deflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:134:(.text.CompressionNative_Deflate+0x2c): undefined reference to `deflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_DeflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:145:(.text.CompressionNative_DeflateEnd+0x29): undefined reference to `deflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateInit2_':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:159:(.text.CompressionNative_InflateInit2_+0x6b): undefined reference to `inflateInit2_'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Inflate':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:171:(.text.CompressionNative_Inflate+0x2c): undefined reference to `inflate'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_InflateEnd':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:182:(.text.CompressionNative_InflateEnd+0x29): undefined reference to `inflateEnd'
/usr/bin/ld.bfd: /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a(pal_zlib.c.o): in function `CompressionNative_Crc32':
/vmr/src/runtime/src/native/libs/System.IO.Compression.Native/pal_zlib.c:192:(.text.CompressionNative_Crc32+0x8): undefined reference to `crc32'
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
/vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/build/Microsoft.NETCore.Native.targets(368,5): error MSB3073: The command ""/usr/bin/clang-14" "/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.o" -o "/vmr/src/runtime/artifacts/bin/crossgen2_publish/x64/Release/native/crossgen2" -Wl,--version-script=/vmr/src/runtime/artifacts/obj/coreclr/crossgen2_publish/linux.x64.Release/native/crossgen2.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libbootstrapper.o /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.ServerGC.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libeventpipe-enabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libRuntime.VxsortEnabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstandalonegc-disabled.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libstdc++compat.a /vmr/src/runtime/artifacts/bin/coreclr/linux.x64.Release/aotsdk/libz.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.IO.Compression.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Net.Security.Native.a /vmr/src/runtime/artifacts/bin/microsoft.netcore.app.runtime.linux-x64/Release/runtimes/linux-x64/native/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/vmr/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

blocking SDK dotnet/sdk#42019. Do we need to set -p:UseSystemZlib=true?
@carlossanlop
carlossanlop
commented
Jul 10, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Yep, I've been notified and I'm looking into this.
Do we need to set -p:UseSystemZlib=true?
No, that property needs to be set conditionally, plus it is already set to true in this failing case, so the problem is not there.
@LoopedBard3
LoopedBard3
commented
Jul 11, 2024

Copy link


Copy Markdown

Member
Likely related improvements:

Linux arm64: dotnet/perf-autofiling-issues#38054
@am11am11
mentioned this pull request
Jul 12, 2024
 Merged


This was referenced Jul 16, 2024

Closed


Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 16, 2024

Open


This was referenced Jul 16, 2024

Closed


Closed

@LoopedBard3LoopedBard3
mentioned this pull request
Jul 18, 2024

Closed

@sebastienros
sebastienros
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
@stephentoub
stephentoub
commented
Jul 19, 2024

Copy link


Copy Markdown

Member
We are seeing different results in the compressed payload in aspnetcore tests, 1 byte difference, is that expected with this switch? Wondering if a gzipped payload is supposed to be deterministic with the same compression level.
There's no guarantee about compressed bytes being exactly the same from version to version. It's expected that updating the zlib implementation may change the exact compressed bytes generated.
@jkurdekjkurdek
mentioned this pull request
Aug 7, 2024

Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Aug 19, 2024
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@ericstjericstjericstj approved these changes
@kgkgAwaiting requested review from kg
@pavelsavarapavelsavaraAwaiting requested review from pavelsavarapavelsavara is a code owner
@directhexdirecthexAwaiting requested review from directhex
@agockeagockeAwaiting requested review from agocke
@jkoritzinskyjkoritzinskyAwaiting requested review from jkoritzinsky
@ilonatommyilonatommyAwaiting requested review from ilonatommy
@lewinglewingAwaiting requested review from lewinglewing is a code owner
@BrzVladBrzVladAwaiting requested review from BrzVladBrzVlad is a code owner
@lambdageeklambdageekAwaiting requested review from lambdageek
@steveisoksteveisokAwaiting requested review from steveisoksteveisok is a code owner
@lateralusXlateralusXAwaiting requested review from lateralusXlateralusX is a code owner
@thaystgthaystgAwaiting requested review from thaystgthaystg is a code owner
@marek-safarmarek-safarAwaiting requested review from marek-safar
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

@carlossanlopcarlossanlop

Labels

area-System.IO.Compression

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

10 participants

@carlossanlop@caaavik-msft@pavelsavara@ericstj@LoopedBard3@am11@sebastienros@stephentoub@jkoritzinsky@jkotas