macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) - #7510

Merged
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp
Oct 1, 2025
Merged

macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload)#7510
ericstj merged 4 commits into
dotnet:mainfrom
asp2286:fix/macos-x64-libomp

Conversation

@asp2286

Copy link
Copy Markdown
Contributor

PR: macOS x64 CI: fix dependency install and OpenMP runtime copy

Summary

This PR fixes MachineLearning-CI failures on macOS x64 where jobs stop at Install MacOS build dependencies with:

Bash exited with code '1'

The breakage comes from two areas:

  1. Dependency install: The pipeline relied on a custom libomp.rb path that no longer works on hosted macOS images.
  2. Helix payload: The script attempted to copy both libomp.dylib and libiomp5.dylib, but libiomp5.dylib is not available when installing libomp from Homebrew core.

Fixes#7509


Changes

build/ci/job-template.yml

  • Replace custom brew install …/build/libomp.rb with standard Homebrew:
    brew update
    brew install -f --overwrite python@3.13
    brew install libomp
    brew link libomp --force
  • Note added: Homebrew ≥4.6 rejects installing formulae from raw paths.

eng/helix.proj

  • macOS x64 only:
    • Set DYLD_LIBRARY_PATH so Helix can find libomp.dylib.
    • Copy only /usr/local/opt/libomp/lib/libomp.dylib into the publish folder.
    • Remove copying of libiomp5.dylib (not present with libomp from Homebrew).
    • Add install-name fix so binaries reference @loader_path/libomp.dylib.

Why

  • Hosted macOS runners changed: raw formula paths are blocked, and only libomp is available via core.
  • Ensures reliable dependency install and payload runtime linking.
  • Other platforms (Linux, Windows, macOS arm64) are unaffected.

Testing

  • Reproduced failure on osx.13.amd64.open queue.
  • With these changes:
    • Dependency install step completes successfully.
    • libomp.dylib is present in publish folder.
    • Helix payload runs with DYLD_LIBRARY_PATH set correctly.
  • Validated in a test run: both macOS x64 Debug/Release proceed past dependency install and build succeeds.

Risk / Impact

  • Low: scoped only to macOS x64 build dependencies and Helix payload.
  • No product code changes, only CI infra adjustments.

Additional Notes

  • Linux and Windows jobs were already green.
  • If maintainers prefer llvm over libomp as the OpenMP provider, happy to adjust.

PR Checklist

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@codecov

codecovBot commented Sep 18, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (1390b39).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7510 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 273999 Branches 28258 28258 ==========================================
- Hits 189093 189075 -18 - Misses 77520 77536 +16 - Partials 7386 7388 +2 
FlagCoverage Δ
Debug69.00% <ø> (-0.01%)⬇️
production63.29% <ø> (-0.01%)⬇️
test89.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 7510 in repo dotnet/machinelearning

@asp2286
asp2286 marked this pull request as draft September 18, 2025 12:03
@asp2286
asp2286 marked this pull request as ready for review September 18, 2025 12:03
@asp2286

Copy link
Copy Markdown
ContributorAuthor

@artl93@ericstj Could you please review this PR? Thank you!

Comment threadbuild/ci/job-template.yml
Comment threadbuild/ci/job-template.yml Outdated
@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch 2 times, most recently from e44de68 to eb0ee80CompareSeptember 28, 2025 16:50
@skyRoma

Copy link
Copy Markdown

I really need these changes, thanks!

Comment threadbuild/ci/job-template.yml Outdated
@ericstj

Copy link
Copy Markdown
Member

I tried to simplify a bit where x64 and M1 were the same. Let's see if this will also resolve the M1 failure.

Comment threadeng/helix.proj
@asp2286

Copy link
Copy Markdown
ContributorAuthor

/ba-g Mac-OS unrelated failures

@asp2286
asp2286force-pushed the fix/macos-x64-libomp branch from f0ce5ba to 1390b39CompareOctober 1, 2025 02:50
@ericstj
ericstj merged commit a5997af into dotnet:mainOct 1, 2025
22 of 25 checks passed
@asp2286
asp2286 deleted the fix/macos-x64-libomp branch October 2, 2025 07:45
ericstj added a commit that referenced this pull request Oct 22, 2025
…ebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
ericstj added a commit that referenced this pull request Oct 23, 2025
* Update dependencies from https://github.com/dotnet/arcade build 20250516.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25266.2
* update cmake mac (#7443)
* update cmake mac
* cmake specific version
* remove specific version
* cmake policy version minimum
* Update build/ci/job-template.yml
* Try adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to libomp.rb
* Update job-template.yml
* Update cmake_minimum_required
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update dependencies from https://github.com/dotnet/arcade build 20250521.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25271.1
* Update dependencies from https://github.com/dotnet/arcade build 20250602.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25302.2
* Update dependencies from https://github.com/dotnet/arcade build 20250625.4
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25325.4
* Update dependencies from https://github.com/dotnet/arcade build 20250716.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25366.1
* Update dependencies from https://github.com/dotnet/arcade build 20250730.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25380.1
* Update dependencies from https://github.com/dotnet/arcade build 20250807.2
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25407.2
* Update dependencies from https://github.com/dotnet/arcade build 20250815.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25415.3
* Update dependencies from https://github.com/dotnet/arcade build 20250828.3
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25428.3
* Update dependencies from https://github.com/dotnet/arcade build 20250910.1
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions
From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25460.1
* Update dependencies from https://github.com/dotnet/arcade build 20250912.4
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25462.4
* Update dependencies from https://github.com/dotnet/arcade build 20250915.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25465.2
* Update dependencies from https://github.com/dotnet/arcade build 20250923.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25473.2
* Update dependencies from https://github.com/dotnet/arcade build 20250925.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25475.3
* Update dependencies from https://github.com/dotnet/arcade build 20251003.3
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25503.3
* Update dependencies from https://github.com/dotnet/arcade build 20251015.2
On relative base path root
Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitExtensions From Version 9.0.0-beta.25111.5 -> To Version 9.0.0-beta.25515.2
* macOS x64 CI: fix dependency install and OpenMP runtime copy (use Homebrew libomp, adjust Helix payload) (#7510)
* CI(macOS x64): use Homebrew libomp + copy only libomp.dylib; remove custom formula path
* Fix indenting and make M1 and x64 consistent
* Simplify helix.proj
---------
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* Update Windows image, fix mac build (#7515)
* Fix new compilation errors with latest SDK
* Fix additional breaks introduced with new SDK
* Update Helix ubuntu arm32 container (#7410)
* Update Helix ubuntu arm32 container
* Switch to debian-12-helix-arm32v7
* Revert back to ubuntu-24.04-helix-arm32v7
* Update helixQueue to use Ubuntu 22.04 on ARM32
* Add linux-arm baseline for Summarize.txt
---------
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Alexey Sokolov <asp2286@gmail.com>
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MacOS_x64 Debug/Release build jobs fail in MachineLearning-CI: “Bash exited with code '1' – Install MacOS build dependencies”

3 participants

@asp2286@skyRoma@ericstj