Uh oh!
There was an error while loading. Please reload this page.
[release/9.0] Fix intermediate artifact upload collisions for NativeAOT and Mono - #125564
Conversation
Multiple platform jobs upload to the same IntermediateArtifacts subdirectory (NativeAOTRuntimePacks or MonoRuntimePacks), causing concurrent write conflicts when arch-independent packages like Microsoft.NETCore.App.Ref are uploaded simultaneously by different platform jobs. Fix by making each upload name unique per platform, matching the pattern already used by CoreCLR ($(osGroup)$(osSubgroup)_$(archType)): - NativeAOT: NativeAOTRuntimePacks_$(osGroup)$(osSubgroup)_$(archType) - Mono: MonoRuntimePacks_$(osGroup)$(osSubgroup)_$(archType) - Mono multithread: MonoRuntimePacks_multithread_$(osGroup)$(osSubgroup)_$(archType) - CrossAOT: MonoRuntimePacks_crossaot_$(osGroup)$(osSubgroup)_$(archType) - LLVM AOT: MonoRuntimePacks_llvmaot_$(osGroup)$(osSubgroup)_$(archType) Updated Workloads download patterns to use MonoRuntimePacks* wildcard to match all per-platform subdirectories. Fixesdotnet#125561 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the release/9.0 runtime-official pipeline to prevent intermittent Azure DevOps artifact upload collisions by making NativeAOT and Mono intermediate artifact subdirectories unique per platform (matching the existing CoreCLR approach).
Changes:
- Make NativeAOT intermediate artifact upload subdirectory unique per platform via
$(osGroup)$(osSubgroup)_$(archType). - Make Mono intermediate artifact upload subdirectory unique per platform, including distinct prefixes for multithread/crossaot/llvmaot variants.
- Update the Workloads job’s
DownloadPipelineArtifactpatterns to useMonoRuntimePacks*so it can find packages across the new per-platform subdirectories.
You can also share your feedback on Copilot code review. Take the survey.
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
steveisok
left a comment
There was a problem hiding this comment.
We get impacted by this problem enough to make it worth a try.
… all macOS/MacCatalyst (#125568) ## Summary macOS 15+ returns `EHOSTUNREACH` instead of `EACCES` for UDP broadcast without `SO_BROADCAST` due to Apple's [Local Network privacy feature](https://developer.apple.com/forums/thread/765285). The test expects `SocketError.AccessDenied` but gets `SocketError.HostUnreachable`. The existing `ActiveIssue` (from #113313) only disabled on MacCatalyst x64. The failure affects all macOS 15+ platforms — confirmed on `osx.15.amd64` in both coreclr and Mono legs in [#125564](#125564). ## Change Broadens the `ActiveIssue` from: ```csharp [ActiveIssue("...", typeof(PlatformDetection), nameof(PlatformDetection.IsMacCatalyst), nameof(PlatformDetection.IsX64Process))] ``` to: ```csharp [ActiveIssue("...", TestPlatforms.OSX | TestPlatforms.MacCatalyst)] ``` @ManickaP requested this backport in #114450 (comment). Fixes#114450 on release/9.0. cc @liveans@kotlarmilos@ManickaP Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes intermittent Azure DevOps artifact upload failures in the release/9.0 official runtime pipeline by ensuring NativeAOT and Mono intermediate artifact uploads use per-platform unique subdirectory names, preventing cross-job blob write collisions for arch-independent packages.
Changes:
- Make NativeAOT intermediate artifact upload names unique per platform using
$(osGroup)$(osSubgroup)_$(archType). - Make Mono intermediate artifact upload names unique per platform (including
multithread,crossaot, andllvmaotvariants). - Update the Workloads job download patterns to use
MonoRuntimePacks*wildcards so it can consume all per-platform Mono upload directories.
You can also share your feedback on Copilot code review. Take the survey.
Uh oh!
There was an error while loading. Please reload this page.
Tagging subscribers to this area: @dotnet/runtime-infrastructure |
lewing
commented
Mar 19, 2026
/ba-g failures are not related |
Uh oh!
There was an error while loading. Please reload this page.
Problem
The
dotnet-runtime-officialpipeline onrelease/9.0has intermittent "File upload failed even after retry" errors caused by multiple platform jobs uploading arch-independent packages (likeMicrosoft.NETCore.App.Ref.9.0.15.nupkg) to the same artifact path simultaneously.Root cause: NativeAOT (27 platforms) all upload to
name: NativeAOTRuntimePacksand Mono (7+ matrix expansions) all upload toname: MonoRuntimePacks— using a shared subdirectory within theIntermediateArtifactsartifact. When two jobs finish at the same time, they race to upload the same file, causing blob storage write conflicts.Compare with CoreCLR, which correctly uses
name: $(osGroup)$(osSubgroup)_$(archType)— unique per platform. ✅Fix
Make each upload name unique per platform (matching the CoreCLR pattern):
NativeAOTRuntimePacksNativeAOTRuntimePacks_$(osGroup)$(osSubgroup)_$(archType)MonoRuntimePacksMonoRuntimePacks_$(osGroup)$(osSubgroup)_$(archType)MonoRuntimePacksMonoRuntimePacks_multithread_$(osGroup)$(osSubgroup)_$(archType)MonoRuntimePacksMonoRuntimePacks_crossaot_$(osGroup)$(osSubgroup)_$(archType)MonoRuntimePacksMonoRuntimePacks_llvmaot_$(osGroup)$(osSubgroup)_$(archType)Updated the Workloads job download patterns from
IntermediateArtifacts/MonoRuntimePacks/Shipping/...toIntermediateArtifacts/MonoRuntimePacks*/Shipping/...so the wildcard matches all per-platform subdirectories.The
CopyFiles@2flatten step (*/Shipping/*.nupkg) still works because each platform subdirectory is still a single path segment.Notes
release/9.0—main/release/10.0have moved to VMR buildsperf-non-wasm-jobs.yml) use the same pattern but only have single-platform uploads (no collision risk), so no changes needed thereFixes#125561
Ref: dotnet/dnceng#1916