Skip to content

Provide prebuilt native libraries by NuGet Packages. - #257

Open
ha-ves wants to merge 103 commits into
NetCordDev:mainfrom
ha-ves:feature/prebuilt-natives
Open

Provide prebuilt native libraries by NuGet Packages.#257
ha-ves wants to merge 103 commits into
NetCordDev:mainfrom
ha-ves:feature/prebuilt-natives

Conversation

@ha-ves

@ha-vesha-ves commented Feb 9, 2026

Copy link
Copy Markdown

Summary

Continues from #245. Implements CI-driven distribution of prebuilt native binary dependencies.

Build NetCord.Natives

  • Download size: ~112 MB
  • Extracted size: ~429 MB

Need to Follow-Up

  • DOCS: instructions for referencing prebuilt packages in projects
  • DOCS: fallback build guide for unsupported platforms or custom RIDs
  • FIX MACOS CI TEST:
dotnet test Tests/NetCord.Natives.Tests/ -tl:off -clp:NoSummary,Verbosity=normal -v normal -bl

and then use https://github.com/JanKrivanek/MSBuildStructuredLog

AllLibraryImportsExistInBinary tests are skipped on OSX due to ProcessModule limitation with dyld.

Included in This PR

✓ CI multi-platform native package build and publish
✓ Per-RID targeted package distribution
✓ Direct integration usage
✓ Custom vcpkg overlay ports for libdave and mlspp &
✓ Dependencies pinning
✓ Basic Unit & Integration tests

Overview

This PR provides prebuilt native libraries via NuGet as per-RID packages. Each package bundles:

  • Runtime binaries for standard .NET consumption
  • Static libraries for NativeAOT compilation
  • MSBuild integration to transparently resolve native paths based on target platform
  • Bundled licenses for all included dependencies

Packages are published automatically on successful CI builds to the configured NuGet feed.

Structure & Packaging Model

Per-RID Distribution Model:

  • Packages are produced per target RID (local or CI builds)
  • Package ID format: NetCord.Natives.{win|linux|osx}-{x64|arm64}
  • Example: NetCord.Natives.linux-x64, NetCord.Natives.win-arm64

Package Contents (per-RID package):

runtimes/{rid}/native/ ← Runtime binaries for standard .NET
staticlibs/{rid}/ ← Static libraries for NativeAOT builds
build/NetCord.Natives.{rid}.* ← MSBuild props/targets per platform
licenses/ ← Copyright files from vcpkg ports

What Changed

CI Pruning:

Core Build & Packaging

  • NetCord.Natives.csproj: VCPKG build and use integration project
  • Custom vcpkg ports (natives-ports/):
    • libdave/portfile.cmake — discord/libdave with MSVC ARM64 patch
    • mlspp/portfile.cmake — cisco/mlspp, libdave's dependency

NuGet Package build files

Accessible Metadata

  • NativesHelper.cs: NativeLibraryVersionAttribute for runtime version discovery of bundled native libs

For issue tracking, refer to nativelibs versions in NuGet package description.

How to Use

Refer to docs included with this PR.

Maintainer Notes: vcpkg Baseline & Dependency Pinning

All native dependencies are pinned to a specific vcpkg baseline (vcpkg.json) to ensure reproducible builds.

Current constraint: openssl==3.0.7

When upgrading native library versions:

  1. Update version constraints in vcpkg.json
  2. Run local vcpkg build to validate against all RID triplets
  3. Verify if licenses are still viable

Resource requirements:

  • Build Runner Availability: On github hosted runner, first-time (cold cache) run can take up to ~30 min

Testing

Comprehensive native library validation via Tests/NetCord.Natives.Tests/:

Unit Tests (NativesBuildTests.cs)

Framework: MSTest with method-level parallelization
Target: .NET 10.0

NativeLoaded — Runtime Library Resolution

[TestMethod][DataRow("libdave")][DataRow("libsodium")][DataRow("opus")][DataRow("zstd")]publicvoidNativeLoaded(stringlibName)
  • Validates each of the four core native libraries can be successfully loaded via NativeLibrary.Load()
  • Catches load failures, missing binaries, or corrupted artifacts

AllLibraryImportsExistInBinary — P/Invoke Symbol Verification

Note: Skipped on OSX due to ProcessModule limitation with dyld preventing export enumeration. NativeAOT static linking tests still validate symbol presence on OSX.

[TestMethod][OSCondition(ConditionMode.Exclude,OperatingSystems.OSX)][DataRow("libdave")][DataRow("libsodium")][DataRow("opus")][DataRow("zstd")]publicvoidAllLibraryImportsExistInBinary(stringlibName,stringclassName)
  • Reflects over managed P/Invoke wrapper classes in the NetCord namespace to extract all [LibraryImport] entry points
  • Loads each native binary and validates every exported symbol exists via NativeLibrary.TryGetExport()
  • Catches missing entry points or mismatched P/Invoke declarations (Update: not possible unless static analysis, not implemented yet)

NativeAotStaticLinking — NativeAOT Full Integration

[TestMethod][DataRow("libdave;libsodium;opus;zstd")]publicvoidNativeAotStaticLinking(stringlibName)
  • Build NativeAot: Invokes dotnet publish -c Release on NativeAotApp with <PublishAot> enabled:
    • Passes current RID (e.g., win-x64, linux-x64, osx-arm64)
    • Links static libraries via <DirectPInvoke>
  • Runtime validation: Executes the published AOT app:
    • All native calls are valid
    • Exit code is 0
    • Logs build and run output for debugging

@ha-ves
ha-vesforce-pushed the feature/prebuilt-natives branch 2 times, most recently from 1c05c69 to 6565a35CompareMay 5, 2026 12:47
@github-actions

Copy link
Copy Markdown

The documentation preview is available at https://preview.netcord.dev/257.

@AraHaan

Copy link
Copy Markdown

I feel like the best option is to do something like:
NetCord.Natives.<rid> that way only the target RID's static libraries are downloaded and used for NAOT and does not slow down the build with extracting unused native static libraries.

@ha-ves
ha-vesforce-pushed the feature/prebuilt-natives branch from 66a1e8d to ee00a64CompareMay 8, 2026 18:10
@ha-ves

Copy link
Copy Markdown
Author

@AraHaan by default CI workflow will provide per-RID packages now.

But by .csproj design it will also work If anyone for some reason still needs to package all in one.

@KubaZ2KubaZ2 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't went through all the MSBuild stuff. I kind of hate that it takes so much and I am afraid it may require changes when new versions of binaries get released. I guess there is no better way to handle that? I think most binaries just require a single make or 2 cmake commands to build, but I think most of the MSBuild stuff is just for copying the binaries in the correct place and licensing?

Comment threadNatives/NetCord.Natives.Tests/NativesBuildTests.cs Outdated
Comment threadNatives/NetCord.Natives/natives-ports/mlspp/portfile.cmake
Comment threadNetCord.Natives/NativesHelper.cs Outdated
Comment threadNetCord.Natives.slnx Outdated
Comment threadTests/NetCord.Natives.Tests/NetCord.Natives.Tests.csproj Outdated
Comment thread.github/workflows/build.yml Outdated
@ha-ves

Copy link
Copy Markdown
Author

@KubaZ2
I haven't went through all the MSBuild stuff. I kind of hate that it takes so much and I am afraid it may require changes when new versions of binaries get released. I guess there is no better way to handle that? I think most binaries just require a single make or 2 cmake commands to build, but I think most of the MSBuild stuff is just for copying the binaries in the correct place and licensing?

Vcpkg is the least time-consuming dependency manager, it can build and put it all together in a convenient path.

The previous version of this PR was just using CMake, but it won't get the dependencies for your NativeAOT.
And it gets many times more complicated than this vcpkg version.

you can refer to this when there's new version:

When upgrading native library versions:

  1. Update version constraints in vcpkg.json (or natives-port// and target specific version and patch)
  2. Run local vcpkg build to validate against all RID triplets
  3. Verify if licenses are still viable

@ha-ves
ha-ves marked this pull request as ready for review May 10, 2026 21:37
@ha-ves

Copy link
Copy Markdown
Author

@KubaZ2, there are some limitations from comments, but I think this PR is ready.

Comment threadDocumentation/guides/basic-concepts/installing-native-dependencies.md Outdated
Comment threadDocumentation/guides/basic-concepts/installing-native-dependencies.md Outdated
Comment threadNetCord.Natives/natives-ports/libdave/vcpkg.json Outdated
Comment threadNetCord.Natives/NetCord.Natives.local.targets Outdated
Comment threadNetCord.Natives/NetCord.Natives.props Outdated
Comment threadNetCord.slnx
Comment threadNatives/Tests/NetCord.Natives.Tests/NativesBuildTests.cs
Comment thread.github/workflows/build-natives.yml Outdated
Comment thread.github/workflows/build-and-publish-natives.yml
Comment thread.github/workflows/build-natives.yml Outdated
Comment threadDocumentation/guides/basic-concepts/installing-native-dependencies.md Outdated
@AraHaan

Copy link
Copy Markdown

I personally think that the native libraries should each have their own packages and that it should be the Discord Bot creator's responsibility to install the native packages themselves when using NAOT. That way those who don't need the native libraries at all (don't use the voice implementation) can continue to get by without the native library packages.

Comment threadTests/NetCord.Natives.Tests/NativeProbes.cs Outdated
CopilotAI added a commit to ha-ves/NetCord that referenced this pull request May 21, 2026
- Fix NetCord.Natives.targets comment typo: 'comes' -> 'come'
- Normalize Importance='High' to 'high' in NetCord.Natives.csproj
- Remove stale commented-out PackageReference in NativeAotApp.csproj
- Update Resources/NuGet/README.md to link to docs for full package list
- Add 'natives' devShell to flake.nix with autoconf/automake/libtool
- Replace macOS brew step with Nix in build-natives.yml; split Build step for Windows/non-Windows
- Pin all GitHub Actions by commit hash in build-natives.yml
- Rename build-publish-natives.yml -> build-and-publish-natives.yml; pin download-artifact hash; fix self-reference in paths trigger
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/f6e9307c-fd0a-40c8-b3a8-e0484782c21e
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
@KubaZ2

KubaZ2 commented May 21, 2026

Copy link
Copy Markdown
Member

I personally think that the native libraries should each have their own packages and that it should be the Discord Bot creator's responsibility to install the native packages themselves when using NAOT. That way those who don't need the native libraries at all (don't use the voice implementation) can continue to get by without the native library packages.

@AraHaan, I am not sure what you mean here. The native packages are completely separate and they are not referenced by NetCord's core package. So it is the creator's responsibility to install the native packages. Not sure what NAOT has to do with it though.

ha-ves added a commit to ha-ves/NetCord that referenced this pull request May 26, 2026
* Implement resolved PR NetCordDev#257 review comments
- Fix NetCord.Natives.targets comment typo: 'comes' -> 'come'
- Normalize Importance='High' to 'high' in NetCord.Natives.csproj
- Remove stale commented-out PackageReference in NativeAotApp.csproj
- Update Resources/NuGet/README.md to link to docs for full package list
- Add 'natives' devShell to flake.nix with autoconf/automake/libtool
- Replace macOS brew step with Nix in build-natives.yml; split Build step for Windows/non-Windows
- Pin all GitHub Actions by commit hash in build-natives.yml
- Rename build-publish-natives.yml -> build-and-publish-natives.yml; pin download-artifact hash; fix self-reference in paths trigger
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/f6e9307c-fd0a-40c8-b3a8-e0484782c21e
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* fix: address natives README link and reflection method visibility
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/e686bf59-4e1e-4ec5-82f8-cc6b3ef93656
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* README: add period to NetCord.Natives table description
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/0553c8bd-a710-4a4d-a8a8-deb3dc917faa
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Deduplicate native build workflow step and remove MSBuild XML schema namespaces
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/a0011395-8ab0-458e-a04e-8152069b7880
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Fix README NetCord.Natives description punctuation
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/a0011395-8ab0-458e-a04e-8152069b7880
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Revert prior README tweak and set NetCord.Natives row text as requested
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/52fb7ee0-c97d-452c-9d94-1eec13cd17ee
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Set README NetCord.Natives row to exact requested text
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/5f1c09c3-27c2-48af-872c-0ae465ec420b
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Update README natives row, native dependencies docs, and test configs
- Update all README.md files with new NetCord.Natives.<RuntimeId> row format
- Replace size column with NuGet package badges in native dependencies table
- Clarify libsodium description regarding voice server encryption requirements
- Update NativeAotApp.csproj: change DirectPInvoke to directly include libraries
- Remove CentrallyManagedPackage false and NoWarn settings from NativeAotApp.csproj
- Remove stray -bl argument from NativesBuildTests.cs GetRunCmd
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/e244793a-3f65-4653-8019-ee2d5151b0c1
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Improve libsodium documentation with clearer explanation of encryption fallback scenarios
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/2a7fe8b8-026b-4918-8d18-9356084ad340
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Fix README for NetCord.Natives
* fix readme in nuget
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
Co-authored-by: Haves Irfan <haves.vansyah@gmail.com>
@ha-ves
ha-ves requested a review from KubaZ2May 26, 2026 14:46
@ha-ves
ha-vesforce-pushed the feature/prebuilt-natives branch 5 times, most recently from bbc14a2 to 1a02cbdCompareMay 26, 2026 17:08
@ha-ves

Copy link
Copy Markdown
Author

@KubaZ2 I can't get nix to work for the workflow ...

We'll have to just use vcpkg for now,
It should be fine since it is the recommended way to build the libraries.

@KubaZ2

Copy link
Copy Markdown
Member

@KubaZ2 I can't get nix to work for the workflow ...

We'll have to just use vcpkg for now,
It should be fine since it is the recommended way to build the libraries.

What exactly was failing?

@ha-ves

Copy link
Copy Markdown
Author

@KubaZ2 I can't get nix to work for the workflow ...
We'll have to just use vcpkg for now,
It should be fine since it is the recommended way to build the libraries.

What exactly was failing?

workflow run available here -> https://github.com/ha-ves/NetCord/actions/runs/26461243098

@KubaZ2

KubaZ2 commented May 28, 2026

Copy link
Copy Markdown
Member

@KubaZ2 I can't get nix to work for the workflow ...
We'll have to just use vcpkg for now,
It should be fine since it is the recommended way to build the libraries.

What exactly was failing?

workflow run available here -> https://github.com/ha-ves/NetCord/actions/runs/26461243098

Seems to be a code or a compiler issue
https://github.com/ha-ves/NetCord/actions/runs/26461243098/job/77909323984#step:9:502

/nix/store/qxaq7jz61a6zkr2mq49i0zvqip2m2jj8-gcc-15.2.0/include/c++/15.2.0/variant(508,22): error GBB9135EA: ‘*(const std::__detail::__variant::_Variant_storage<false, mlspp::Add, mlspp::Update, mlspp::Remove, mlspp::PreSharedKey, mlspp::ReInit, mlspp::ExternalInit, mlspp::GroupContextExtensions>*)__result.std::__detail::__variant::_Variant_storage<false, mlspp::Add, mlspp::Update, mlspp::Remove, mlspp::PreSharedKey, mlspp::ReInit, mlspp::ExternalInit, mlspp::GroupContextExtensions>::_M_index’ may be used uninitialized [-Werror=maybe-uninitialized] [/home/runner/work/NetCord/NetCord/NetCord.Natives/NetCord.Natives.csproj]
508 | return this->_M_index != __index_type(variant_npos);
| ~~~~~~^~~~~~~~
cc1plus: all warnings being treated as errors

ha-ves pushed a commit to ha-ves/NetCord that referenced this pull request May 30, 2026
* Implement resolved PR NetCordDev#257 review comments
- Fix NetCord.Natives.targets comment typo: 'comes' -> 'come'
- Normalize Importance='High' to 'high' in NetCord.Natives.csproj
- Remove stale commented-out PackageReference in NativeAotApp.csproj
- Update Resources/NuGet/README.md to link to docs for full package list
- Add 'natives' devShell to flake.nix with autoconf/automake/libtool
- Replace macOS brew step with Nix in build-natives.yml; split Build step for Windows/non-Windows
- Pin all GitHub Actions by commit hash in build-natives.yml
- Rename build-publish-natives.yml -> build-and-publish-natives.yml; pin download-artifact hash; fix self-reference in paths trigger
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/f6e9307c-fd0a-40c8-b3a8-e0484782c21e
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* fix: address natives README link and reflection method visibility
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/e686bf59-4e1e-4ec5-82f8-cc6b3ef93656
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* README: add period to NetCord.Natives table description
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/0553c8bd-a710-4a4d-a8a8-deb3dc917faa
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Deduplicate native build workflow step and remove MSBuild XML schema namespaces
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/a0011395-8ab0-458e-a04e-8152069b7880
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Fix README NetCord.Natives description punctuation
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/a0011395-8ab0-458e-a04e-8152069b7880
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Revert prior README tweak and set NetCord.Natives row text as requested
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/52fb7ee0-c97d-452c-9d94-1eec13cd17ee
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Set README NetCord.Natives row to exact requested text
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/5f1c09c3-27c2-48af-872c-0ae465ec420b
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Update README natives row, native dependencies docs, and test configs
- Update all README.md files with new NetCord.Natives.<RuntimeId> row format
- Replace size column with NuGet package badges in native dependencies table
- Clarify libsodium description regarding voice server encryption requirements
- Update NativeAotApp.csproj: change DirectPInvoke to directly include libraries
- Remove CentrallyManagedPackage false and NoWarn settings from NativeAotApp.csproj
- Remove stray -bl argument from NativesBuildTests.cs GetRunCmd
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/e244793a-3f65-4653-8019-ee2d5151b0c1
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Improve libsodium documentation with clearer explanation of encryption fallback scenarios
Agent-Logs-Url: https://github.com/ha-ves/NetCord/sessions/2a7fe8b8-026b-4918-8d18-9356084ad340
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
* Fix README for NetCord.Natives
* fix readme in nuget
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ha-ves <20160532+ha-ves@users.noreply.github.com>
Co-authored-by: Haves Irfan <haves.vansyah@gmail.com>
fix CI shell error
@ha-ves
ha-vesforce-pushed the feature/prebuilt-natives branch from 1a02cbd to fac897fCompareMay 30, 2026 15:41
Comment thread.github/workflows/build-natives-windows.yml Outdated
Comment threadREADME.md Outdated
@ha-ves
ha-vesforce-pushed the feature/prebuilt-natives branch from bb1d06c to 4465edaCompareJuly 4, 2026 18:01
ha-vesand others added 29 commits July 5, 2026 03:12
* custom triplets
* proper msbuild flow
* revert symlink to be combined in main build files
* proper symlink and prevent copy to output
* fix musl builtrid path
* windows path length check
* nativeaotstaticlinking test remove dll check
* CI & whitespace cleanup
* fix vcpkgroot defaults to submodule path
* remove bootstrap vcpkg target for non windows
* fix mlspp vcpkg portfile for clang/gcc flags
* reduce packaged build files
* update docs
* fix natives meta-package
* fix build-natives.yml typo
* fix build files redundancies & breaking changes
* add github package publish
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@ha-ves@AraHaan@KubaZ2