Skip to content

GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243

Merged
kou merged 8 commits into
apache:mainfrom
tadeja:verify-release-macOS
Jun 29, 2026
Merged

GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom
tadeja:verify-release-macOS

Conversation

@tadeja

@tadejatadeja commented Jun 23, 2026•
edited
Loading

Copy link
Copy Markdown
Collaborator

Rationale for this change

The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).
Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.
The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.

What changes are included in this PR?

Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.
(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)

Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).

Are these changes tested?

Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.

Are there any user-facing changes?

No.

@github-actions

github-actionsBot commented Jun 23, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.

@github-actionsgithub-actionsBot added the awaiting review Awaiting review label Jun 23, 2026
@tadeja

tadeja commented Jun 23, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64

@github-actions

github-actionsBot commented Jun 23, 2026

Copy link
Copy Markdown

Revision: 08323e7

Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2

TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions

@tadeja

tadeja commented Jun 23, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64

@github-actions

github-actionsBot commented Jun 23, 2026

Copy link
Copy Markdown

Revision: 08323e7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004

TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions

@tadeja

tadeja commented Jun 24, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@github-actions crossbow submit -g verify-rc-source-macos

@github-actions

github-actionsBot commented Jun 24, 2026

Copy link
Copy Markdown

Revision: 0bc831e

Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce

TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions

@tadeja

tadeja commented Jun 24, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@github-actions crossbow submit -g verify-rc-source-macos

@github-actions

github-actionsBot commented Jun 24, 2026

Copy link
Copy Markdown

Revision: 6a8426d

Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b

TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions

@tadejatadeja changed the title GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja

tadeja commented Jun 24, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64

@github-actions

github-actionsBot commented Jun 24, 2026

Copy link
Copy Markdown

Revision: 8f573bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3

TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions

@tadejatadeja changed the title GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja marked this pull request as ready for review June 25, 2026 10:55
@tadejatadeja mentioned this pull request Jun 25, 2026
Closed
@tadejatadeja changed the title GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed the verify-release-macOS branch from 8f1149d to 800380eCompareJune 26, 2026 17:43
kou
kou approved these changes Jun 26, 2026
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Comment threaddev/release/verify-release-candidate.sh Outdated
# path (searched last, to not override system libs) so they resolve.
# See https://github.com/conda-forge/cmake-feedstock/issues/230
if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install

@koukouJun 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:

 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Jun 26, 2026
@tadeja

tadeja commented Jun 27, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64

@github-actions

github-actionsBot commented Jun 27, 2026

Copy link
Copy Markdown

Revision: c24bafa

Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f

TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions

@kou
kou merged commit 4f60019 into apache:mainJun 29, 2026
8 checks passed
@koukou removed the awaiting merge Awaiting merge label Jun 29, 2026
@conbench-apache-arrow

conbench-apache-arrowBot commented Jun 29, 2026

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.

@tadeja
tadeja deleted the verify-release-macOS branch July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build by tadeja · Pull Request #50243 · apache/arrow · GitHub
Skip to content
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243
Merged
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Jun 29, 2026
Merged
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Conversation
@tadeja
@tadejatadeja
commented
Jun 23, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Rationale for this change
The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.
What changes are included in this PR?
Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.

(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)
Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).
Are these changes tested?
Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.
Are there any user-facing changes?
No.
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jun 23, 2026
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2
TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 0bc831e
Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 6a8426d
Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 8f573bb
Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja
marked this pull request as ready for review
June 25, 2026 10:55
@tadejatadeja
mentioned this pull request
Jun 25, 2026

Closed

@tadejatadeja
changed the title
GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed
the
verify-release-macOS
branch
from
8f1149d to
800380eCompareJune 26, 2026 17:43
kou
kou
approved these changes
Jun 26, 2026
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/release/verify-release-candidate.sh
Outdated

 # path (searched last, to not override system libs) so they resolve.
 # See https://github.com/conda-forge/cmake-feedstock/issues/230
 if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
 DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install
@koukouJun 26, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:
 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting review
Awaiting review
labels
Jun 26, 2026
@tadeja
tadeja
commented
Jun 27, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 27, 2026

Copy link


Copy Markdown

Revision: c24bafa
Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@kou
kou
merged commit 4f60019
into
apache:mainJun 29, 2026

8 checks passed

@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jun 29, 2026
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jun 29, 2026

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
@tadeja
tadeja
deleted the

verify-release-macOS

branch
July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build by tadeja · Pull Request #50243 · apache/arrow · GitHub
Skip to content
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243
Merged
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Jun 29, 2026
Merged
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Conversation
@tadeja
@tadejatadeja
commented
Jun 23, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Rationale for this change
The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.
What changes are included in this PR?
Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.

(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)
Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).
Are these changes tested?
Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.
Are there any user-facing changes?
No.
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jun 23, 2026
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2
TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 0bc831e
Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 6a8426d
Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 8f573bb
Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja
marked this pull request as ready for review
June 25, 2026 10:55
@tadejatadeja
mentioned this pull request
Jun 25, 2026

Closed

@tadejatadeja
changed the title
GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed
the
verify-release-macOS
branch
from
8f1149d to
800380eCompareJune 26, 2026 17:43
kou
kou
approved these changes
Jun 26, 2026
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/release/verify-release-candidate.sh
Outdated

 # path (searched last, to not override system libs) so they resolve.
 # See https://github.com/conda-forge/cmake-feedstock/issues/230
 if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
 DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install
@koukouJun 26, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:
 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting review
Awaiting review
labels
Jun 26, 2026
@tadeja
tadeja
commented
Jun 27, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 27, 2026

Copy link


Copy Markdown

Revision: c24bafa
Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@kou
kou
merged commit 4f60019
into
apache:mainJun 29, 2026

8 checks passed

@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jun 29, 2026
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jun 29, 2026

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
@tadeja
tadeja
deleted the

verify-release-macOS

branch
July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build by tadeja · Pull Request #50243 · apache/arrow · GitHub
Skip to content
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243
Merged
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Jun 29, 2026
Merged
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Conversation
@tadeja
@tadejatadeja
commented
Jun 23, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Rationale for this change
The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.
What changes are included in this PR?
Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.

(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)
Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).
Are these changes tested?
Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.
Are there any user-facing changes?
No.
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jun 23, 2026
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2
TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 0bc831e
Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 6a8426d
Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 8f573bb
Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja
marked this pull request as ready for review
June 25, 2026 10:55
@tadejatadeja
mentioned this pull request
Jun 25, 2026

Closed

@tadejatadeja
changed the title
GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed
the
verify-release-macOS
branch
from
8f1149d to
800380eCompareJune 26, 2026 17:43
kou
kou
approved these changes
Jun 26, 2026
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/release/verify-release-candidate.sh
Outdated

 # path (searched last, to not override system libs) so they resolve.
 # See https://github.com/conda-forge/cmake-feedstock/issues/230
 if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
 DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install
@koukouJun 26, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:
 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting review
Awaiting review
labels
Jun 26, 2026
@tadeja
tadeja
commented
Jun 27, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 27, 2026

Copy link


Copy Markdown

Revision: c24bafa
Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@kou
kou
merged commit 4f60019
into
apache:mainJun 29, 2026

8 checks passed

@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jun 29, 2026
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jun 29, 2026

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
@tadeja
tadeja
deleted the

verify-release-macOS

branch
July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build by tadeja · Pull Request #50243 · apache/arrow · GitHub
Skip to content
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243
Merged
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Jun 29, 2026
Merged
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Conversation
@tadeja
@tadejatadeja
commented
Jun 23, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Rationale for this change
The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.
What changes are included in this PR?
Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.

(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)
Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).
Are these changes tested?
Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.
Are there any user-facing changes?
No.
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jun 23, 2026
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2
TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 0bc831e
Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 6a8426d
Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 8f573bb
Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja
marked this pull request as ready for review
June 25, 2026 10:55
@tadejatadeja
mentioned this pull request
Jun 25, 2026

Closed

@tadejatadeja
changed the title
GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed
the
verify-release-macOS
branch
from
8f1149d to
800380eCompareJune 26, 2026 17:43
kou
kou
approved these changes
Jun 26, 2026
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/release/verify-release-candidate.sh
Outdated

 # path (searched last, to not override system libs) so they resolve.
 # See https://github.com/conda-forge/cmake-feedstock/issues/230
 if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
 DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install
@koukouJun 26, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:
 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting review
Awaiting review
labels
Jun 26, 2026
@tadeja
tadeja
commented
Jun 27, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 27, 2026

Copy link


Copy Markdown

Revision: c24bafa
Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@kou
kou
merged commit 4f60019
into
apache:mainJun 29, 2026

8 checks passed

@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jun 29, 2026
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jun 29, 2026

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
@tadeja
tadeja
deleted the

verify-release-macOS

branch
July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build by tadeja · Pull Request #50243 · apache/arrow · GitHub
Skip to content
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243
Merged
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Jun 29, 2026
Merged
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Conversation
@tadeja
@tadejatadeja
commented
Jun 23, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Rationale for this change
The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.
What changes are included in this PR?
Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.

(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)
Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).
Are these changes tested?
Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.
Are there any user-facing changes?
No.
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jun 23, 2026
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2
TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 0bc831e
Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 6a8426d
Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 8f573bb
Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja
marked this pull request as ready for review
June 25, 2026 10:55
@tadejatadeja
mentioned this pull request
Jun 25, 2026

Closed

@tadejatadeja
changed the title
GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed
the
verify-release-macOS
branch
from
8f1149d to
800380eCompareJune 26, 2026 17:43
kou
kou
approved these changes
Jun 26, 2026
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/release/verify-release-candidate.sh
Outdated

 # path (searched last, to not override system libs) so they resolve.
 # See https://github.com/conda-forge/cmake-feedstock/issues/230
 if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
 DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install
@koukouJun 26, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:
 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting review
Awaiting review
labels
Jun 26, 2026
@tadeja
tadeja
commented
Jun 27, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 27, 2026

Copy link


Copy Markdown

Revision: c24bafa
Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@kou
kou
merged commit 4f60019
into
apache:mainJun 29, 2026

8 checks passed

@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jun 29, 2026
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jun 29, 2026

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
@tadeja
tadeja
deleted the

verify-release-macOS

branch
July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build by tadeja · Pull Request #50243 · apache/arrow · GitHub
Skip to content
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243
Merged
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Jun 29, 2026
Merged
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Conversation
@tadeja
@tadejatadeja
commented
Jun 23, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Rationale for this change
The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.
What changes are included in this PR?
Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.

(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)
Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).
Are these changes tested?
Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.
Are there any user-facing changes?
No.
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jun 23, 2026
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2
TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 0bc831e
Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 6a8426d
Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 8f573bb
Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja
marked this pull request as ready for review
June 25, 2026 10:55
@tadejatadeja
mentioned this pull request
Jun 25, 2026

Closed

@tadejatadeja
changed the title
GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed
the
verify-release-macOS
branch
from
8f1149d to
800380eCompareJune 26, 2026 17:43
kou
kou
approved these changes
Jun 26, 2026
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/release/verify-release-candidate.sh
Outdated

 # path (searched last, to not override system libs) so they resolve.
 # See https://github.com/conda-forge/cmake-feedstock/issues/230
 if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
 DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install
@koukouJun 26, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:
 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting review
Awaiting review
labels
Jun 26, 2026
@tadeja
tadeja
commented
Jun 27, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 27, 2026

Copy link


Copy Markdown

Revision: c24bafa
Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@kou
kou
merged commit 4f60019
into
apache:mainJun 29, 2026

8 checks passed

@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jun 29, 2026
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jun 29, 2026

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
@tadeja
tadeja
deleted the

verify-release-macOS

branch
July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build by tadeja · Pull Request #50243 · apache/arrow · GitHub
Skip to content
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build - #50243
Merged
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Jun 29, 2026
Merged
GH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc build#50243
kou merged 8 commits into
apache:mainfrom 
tadeja:verify-release-macOS
Conversation
@tadeja
@tadejatadeja
commented
Jun 23, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Rationale for this change
The nightly job Verify release candidate on macOS intermittently aborts the C++ build with Library not loaded: @rpath/libLLVM.22.1.dylib).

Conda's LLVM tools resolve libLLVM via @loader_path/../lib and can point at the package cache which has no libLLVM. The conda env lib dir is not on the loader path.

The same @rpath miss seen with three different conda libraries (libLLVM, libprotoc, libabsl) through four binaries (llvm-link and llvm-ranlib, grpc_cpp_plugin, protoc) and may affect any shared library on osx - conda-forge/cmake-feedstock#230.
What changes are included in this PR?
Add $CONDA_PREFIX/lib to DYLD_FALLBACK_LIBRARY_PATH for the step cmake --build.

(Fallback is searched last, so it resolves the @rpath miss without overriding system libraries.)
Also drop unneeded empty element from DYLD_LIBRARY_PATH/LD_LIBRARY_PATH assignment (trailing : when the variable was unset).
Are these changes tested?
Tested by nightly jobs verify-rc-source--macos-conda- : cpp, integration and python tasks pass.
Are there any user-facing changes?
No.
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

⚠️ GitHub issue #50242has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jun 23, 2026
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-b28c2cbbb2
TaskStatus
verify-rc-source-integration-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 23, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-cpp-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 23, 2026

Copy link


Copy Markdown

Revision: 08323e7
Submitted crossbow builds: ursacomputing/crossbow @ actions-d35059f004
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 0bc831e
Submitted crossbow builds: ursacomputing/crossbow @ actions-7b92c7f5ce
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit -g verify-rc-source-macos
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 6a8426d
Submitted crossbow builds: ursacomputing/crossbow @ actions-aa38f94d3b
TaskStatus
verify-rc-source-cpp-macos-amd64GitHub Actions
verify-rc-source-cpp-macos-arm64GitHub Actions
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-amd64GitHub Actions
verify-rc-source-integration-macos-arm64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-amd64GitHub Actions
verify-rc-source-python-macos-arm64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
verify-rc-source-ruby-macos-amd64GitHub Actions
verify-rc-source-ruby-macos-arm64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [C++][Release] Pin LLVM <22 in conda env to avoid LLVM 22.1.8 on macOSGH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 24, 2026
@tadeja
tadeja
commented
Jun 24, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-integration-macos-conda-amd64 verify-rc-source-cpp-macos-conda-amd64 verify-rc-source-python-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 24, 2026

Copy link


Copy Markdown

Revision: 8f573bb
Submitted crossbow builds: ursacomputing/crossbow @ actions-c854bae6e3
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@tadejatadeja
changed the title
GH-50242: [Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildJun 25, 2026
@tadeja
tadeja
marked this pull request as ready for review
June 25, 2026 10:55
@tadejatadeja
mentioned this pull request
Jun 25, 2026

Closed

@tadejatadeja
changed the title
GH-50242: [CI][Release] Fix flaky libLLVM load failure in macOS verify-rc buildGH-50242: [CI][Release] Fix flaky dylib load failure in macOS verify-rc buildJun 26, 2026
@tadeja
tadejaforce-pushed
the
verify-release-macOS
branch
from
8f1149d to
800380eCompareJune 26, 2026 17:43
kou
kou
approved these changes
Jun 26, 2026
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/release/verify-release-candidate.sh
Outdated

 # path (searched last, to not override system libs) so they resolve.
 # See https://github.com/conda-forge/cmake-feedstock/issues/230
 if [ "$(uname)" = "Darwin" ] && [ "${USE_CONDA}" -gt 0 ] && [ -n "${CONDA_PREFIX:-}" ]; then
 DYLD_FALLBACK_LIBRARY_PATH="${CONDA_PREFIX}/lib:${DYLD_FALLBACK_LIBRARY_PATH:-/usr/local/lib:/usr/lib}" cmake --build . --target install
@koukouJun 26, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we don't need the default value. Here is man dyld on my Mac mini:
 DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories that contain libraries. If a dylib is
not found at its install path, dyld uses this as a list of directories to search for
the dylib.
For new binaries (Fall 2023 or later) there is no default. For older binaries, there
is a default fallback search path of: /usr/local/lib:/usr/lib.

@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting review
Awaiting review
labels
Jun 26, 2026
@tadeja
tadeja
commented
Jun 27, 2026

Copy link


Copy Markdown

CollaboratorAuthor
@github-actions crossbow submit verify-rc-source-*-macos-conda-amd64
@github-actions
github-actionsBot
commented
Jun 27, 2026

Copy link


Copy Markdown

Revision: c24bafa
Submitted crossbow builds: ursacomputing/crossbow @ actions-eaa3774d9f
TaskStatus
verify-rc-source-cpp-macos-conda-amd64GitHub Actions
verify-rc-source-integration-macos-conda-amd64GitHub Actions
verify-rc-source-python-macos-conda-amd64GitHub Actions
@kou
kou
merged commit 4f60019
into
apache:mainJun 29, 2026

8 checks passed

@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jun 29, 2026
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jun 29, 2026

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 4f60019.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
@tadeja
tadeja
deleted the

verify-release-macOS

branch
July 23, 2026 17:34
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@jonkeanejonkeaneAwaiting requested review from jonkeanejonkeane is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@tadeja@kou