GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336

Merged
kou merged 2 commits into
apache:mainfrom
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged

GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom
kou:cpp-macos-gandiva-openssl

Conversation

@kou

@koukou commented Jun 28, 2023•
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Member

Rationale for this change

GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.

If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.

What changes are included in this PR?

This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.

Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions

github-actionsBot commented Jun 28, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.

@kou

kou commented Jun 28, 2023

Copy link
Copy Markdown
MemberAuthor

+1

The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .

pitrou
pitrou reviewed Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake Outdated
OUTPUT_STRIP_TRAILING_WHITESPACE)
if(OPENSSL_BREW_PREFIX)
set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
if(OPENSSL11_BREW_PREFIX)

@pitroupitrouJun 28, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?

@koukouJun 29, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!

Ah, wait. I should have used OPENSSL30_... for it.

pitrou
pitrou reviewed Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake Outdated
if(BREW)
execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
execute_process(COMMAND ${BREW} --prefix "openssl"

@pitroupitrouJun 28, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?

Wouldn't "openssl" cover all other cases? I don't know how brew works...

@koukouJun 29, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).
If openssl@3.1 isn't installed, brew --prefix openssl is failed.

Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.

@raulcd

raulcd commented Jun 28, 2023

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@raulcd

raulcd commented Jun 28, 2023

Copy link
Copy Markdown
Member

Triggering java-jars because they seem to be failing on the nightlies due to this reason:

 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions

github-actionsBot commented Jun 28, 2023

Copy link
Copy Markdown

Revision: fc0a16e

Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54

TaskStatus
java-jarsGithub Actions

@raulcd

raulcd commented Jun 28, 2023

Copy link
Copy Markdown
Member

There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:

 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jun 29, 2023
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 29, 2023
@kou

kou commented Jun 29, 2023

Copy link
Copy Markdown
MemberAuthor

gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.

Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.

@raulcd

raulcd commented Jun 29, 2023

Copy link
Copy Markdown
Member

Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.

Sounds good to me.

@kou

kou commented Jun 30, 2023

Copy link
Copy Markdown
MemberAuthor

Created: #36404

I'll merge this.

@kou
kou merged commit 9d92ed4 into apache:mainJun 30, 2023
@kou
kou deleted the cpp-macos-gandiva-openssl branch June 30, 2023 00:41
@koukou removed the awaiting change review Awaiting change review label Jun 30, 2023
@paleolimbotpaleolimbot mentioned this pull request Jul 4, 2023
Closed
@conbench-apache-arrow

conbench-apache-arrowBot commented Jul 5, 2023

Copy link
Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.

There were 5 benchmark results indicating a performance regression:

The full Conbench report has more details.

lriggs pushed a commit to lriggs/arrow that referenced this pull request Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit to dremio/arrow that referenced this pull request Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit to dremio/arrow that referenced this pull request Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS

3 participants

@kou@raulcd@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336
Merged
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged
GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Conversation
@kou
@koukou
commented
Jun 28, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
GitHub Actions self-hosted runner for macOS has

/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.
@kou
kou
commented
Jun 28, 2023

Copy link


Copy Markdown

MemberAuthor
+1
The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 OUTPUT_STRIP_TRAILING_WHITESPACE)
 if(OPENSSL_BREW_PREFIX)
 set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
 if(OPENSSL11_BREW_PREFIX)
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!
Ah, wait. I should have used OPENSSL30_... for it.
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 if(BREW)
 execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
 OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
 execute_process(COMMAND ${BREW} --prefix "openssl"
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?
Wouldn't "openssl" cover all other cases? I don't know how brew works...
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).

If openssl@3.1 isn't installed, brew --prefix openssl is failed.
Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
Triggering java-jars because they seem to be failing on the nightlies due to this reason:
 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

Revision: fc0a16e
Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54
TaskStatus
java-jarsGithub Actions
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:
 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
Jun 29, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Jun 29, 2023
@kou
kou
commented
Jun 29, 2023

Copy link


Copy Markdown

MemberAuthor
gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
@raulcd
raulcd
commented
Jun 29, 2023

Copy link


Copy Markdown

Member
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
Sounds good to me.
@kou
kou
commented
Jun 30, 2023

Copy link


Copy Markdown

MemberAuthor
Created: #36404
I'll merge this.
@kou
kou
merged commit 9d92ed4
into
apache:mainJun 30, 2023
@kou
kou
deleted the

cpp-macos-gandiva-openssl

branch
June 30, 2023 00:41
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Jun 30, 2023
@paleolimbotpaleolimbot
mentioned this pull request
Jul 4, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 5, 2023

Copy link


Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.
There were 5 benchmark results indicating a performance regression:
The full Conbench report has more details.
lriggs pushed a commit
to lriggs/arrow
that referenced
this pull request
Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit
to dremio/arrow
that referenced
this pull request
Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit
to dremio/arrow
that referenced
this pull request
Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS


3 participants

@kou@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336
Merged
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged
GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Conversation
@kou
@koukou
commented
Jun 28, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
GitHub Actions self-hosted runner for macOS has

/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.
@kou
kou
commented
Jun 28, 2023

Copy link


Copy Markdown

MemberAuthor
+1
The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 OUTPUT_STRIP_TRAILING_WHITESPACE)
 if(OPENSSL_BREW_PREFIX)
 set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
 if(OPENSSL11_BREW_PREFIX)
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!
Ah, wait. I should have used OPENSSL30_... for it.
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 if(BREW)
 execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
 OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
 execute_process(COMMAND ${BREW} --prefix "openssl"
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?
Wouldn't "openssl" cover all other cases? I don't know how brew works...
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).

If openssl@3.1 isn't installed, brew --prefix openssl is failed.
Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
Triggering java-jars because they seem to be failing on the nightlies due to this reason:
 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

Revision: fc0a16e
Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54
TaskStatus
java-jarsGithub Actions
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:
 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
Jun 29, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Jun 29, 2023
@kou
kou
commented
Jun 29, 2023

Copy link


Copy Markdown

MemberAuthor
gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
@raulcd
raulcd
commented
Jun 29, 2023

Copy link


Copy Markdown

Member
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
Sounds good to me.
@kou
kou
commented
Jun 30, 2023

Copy link


Copy Markdown

MemberAuthor
Created: #36404
I'll merge this.
@kou
kou
merged commit 9d92ed4
into
apache:mainJun 30, 2023
@kou
kou
deleted the

cpp-macos-gandiva-openssl

branch
June 30, 2023 00:41
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Jun 30, 2023
@paleolimbotpaleolimbot
mentioned this pull request
Jul 4, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 5, 2023

Copy link


Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.
There were 5 benchmark results indicating a performance regression:
The full Conbench report has more details.
lriggs pushed a commit
to lriggs/arrow
that referenced
this pull request
Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit
to dremio/arrow
that referenced
this pull request
Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit
to dremio/arrow
that referenced
this pull request
Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS


3 participants

@kou@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336
Merged
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged
GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Conversation
@kou
@koukou
commented
Jun 28, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
GitHub Actions self-hosted runner for macOS has

/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.
@kou
kou
commented
Jun 28, 2023

Copy link


Copy Markdown

MemberAuthor
+1
The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 OUTPUT_STRIP_TRAILING_WHITESPACE)
 if(OPENSSL_BREW_PREFIX)
 set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
 if(OPENSSL11_BREW_PREFIX)
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!
Ah, wait. I should have used OPENSSL30_... for it.
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 if(BREW)
 execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
 OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
 execute_process(COMMAND ${BREW} --prefix "openssl"
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?
Wouldn't "openssl" cover all other cases? I don't know how brew works...
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).

If openssl@3.1 isn't installed, brew --prefix openssl is failed.
Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
Triggering java-jars because they seem to be failing on the nightlies due to this reason:
 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

Revision: fc0a16e
Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54
TaskStatus
java-jarsGithub Actions
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:
 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
Jun 29, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Jun 29, 2023
@kou
kou
commented
Jun 29, 2023

Copy link


Copy Markdown

MemberAuthor
gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
@raulcd
raulcd
commented
Jun 29, 2023

Copy link


Copy Markdown

Member
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
Sounds good to me.
@kou
kou
commented
Jun 30, 2023

Copy link


Copy Markdown

MemberAuthor
Created: #36404
I'll merge this.
@kou
kou
merged commit 9d92ed4
into
apache:mainJun 30, 2023
@kou
kou
deleted the

cpp-macos-gandiva-openssl

branch
June 30, 2023 00:41
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Jun 30, 2023
@paleolimbotpaleolimbot
mentioned this pull request
Jul 4, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 5, 2023

Copy link


Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.
There were 5 benchmark results indicating a performance regression:
The full Conbench report has more details.
lriggs pushed a commit
to lriggs/arrow
that referenced
this pull request
Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit
to dremio/arrow
that referenced
this pull request
Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit
to dremio/arrow
that referenced
this pull request
Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS


3 participants

@kou@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336
Merged
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged
GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Conversation
@kou
@koukou
commented
Jun 28, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
GitHub Actions self-hosted runner for macOS has

/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.
@kou
kou
commented
Jun 28, 2023

Copy link


Copy Markdown

MemberAuthor
+1
The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 OUTPUT_STRIP_TRAILING_WHITESPACE)
 if(OPENSSL_BREW_PREFIX)
 set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
 if(OPENSSL11_BREW_PREFIX)
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!
Ah, wait. I should have used OPENSSL30_... for it.
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 if(BREW)
 execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
 OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
 execute_process(COMMAND ${BREW} --prefix "openssl"
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?
Wouldn't "openssl" cover all other cases? I don't know how brew works...
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).

If openssl@3.1 isn't installed, brew --prefix openssl is failed.
Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
Triggering java-jars because they seem to be failing on the nightlies due to this reason:
 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

Revision: fc0a16e
Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54
TaskStatus
java-jarsGithub Actions
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:
 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
Jun 29, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Jun 29, 2023
@kou
kou
commented
Jun 29, 2023

Copy link


Copy Markdown

MemberAuthor
gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
@raulcd
raulcd
commented
Jun 29, 2023

Copy link


Copy Markdown

Member
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
Sounds good to me.
@kou
kou
commented
Jun 30, 2023

Copy link


Copy Markdown

MemberAuthor
Created: #36404
I'll merge this.
@kou
kou
merged commit 9d92ed4
into
apache:mainJun 30, 2023
@kou
kou
deleted the

cpp-macos-gandiva-openssl

branch
June 30, 2023 00:41
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Jun 30, 2023
@paleolimbotpaleolimbot
mentioned this pull request
Jul 4, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 5, 2023

Copy link


Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.
There were 5 benchmark results indicating a performance regression:
The full Conbench report has more details.
lriggs pushed a commit
to lriggs/arrow
that referenced
this pull request
Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit
to dremio/arrow
that referenced
this pull request
Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit
to dremio/arrow
that referenced
this pull request
Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS


3 participants

@kou@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336
Merged
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged
GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Conversation
@kou
@koukou
commented
Jun 28, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
GitHub Actions self-hosted runner for macOS has

/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.
@kou
kou
commented
Jun 28, 2023

Copy link


Copy Markdown

MemberAuthor
+1
The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 OUTPUT_STRIP_TRAILING_WHITESPACE)
 if(OPENSSL_BREW_PREFIX)
 set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
 if(OPENSSL11_BREW_PREFIX)
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!
Ah, wait. I should have used OPENSSL30_... for it.
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 if(BREW)
 execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
 OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
 execute_process(COMMAND ${BREW} --prefix "openssl"
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?
Wouldn't "openssl" cover all other cases? I don't know how brew works...
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).

If openssl@3.1 isn't installed, brew --prefix openssl is failed.
Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
Triggering java-jars because they seem to be failing on the nightlies due to this reason:
 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

Revision: fc0a16e
Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54
TaskStatus
java-jarsGithub Actions
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:
 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
Jun 29, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Jun 29, 2023
@kou
kou
commented
Jun 29, 2023

Copy link


Copy Markdown

MemberAuthor
gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
@raulcd
raulcd
commented
Jun 29, 2023

Copy link


Copy Markdown

Member
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
Sounds good to me.
@kou
kou
commented
Jun 30, 2023

Copy link


Copy Markdown

MemberAuthor
Created: #36404
I'll merge this.
@kou
kou
merged commit 9d92ed4
into
apache:mainJun 30, 2023
@kou
kou
deleted the

cpp-macos-gandiva-openssl

branch
June 30, 2023 00:41
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Jun 30, 2023
@paleolimbotpaleolimbot
mentioned this pull request
Jul 4, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 5, 2023

Copy link


Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.
There were 5 benchmark results indicating a performance regression:
The full Conbench report has more details.
lriggs pushed a commit
to lriggs/arrow
that referenced
this pull request
Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit
to dremio/arrow
that referenced
this pull request
Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit
to dremio/arrow
that referenced
this pull request
Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS


3 participants

@kou@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336
Merged
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged
GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Conversation
@kou
@koukou
commented
Jun 28, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
GitHub Actions self-hosted runner for macOS has

/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.
@kou
kou
commented
Jun 28, 2023

Copy link


Copy Markdown

MemberAuthor
+1
The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 OUTPUT_STRIP_TRAILING_WHITESPACE)
 if(OPENSSL_BREW_PREFIX)
 set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
 if(OPENSSL11_BREW_PREFIX)
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!
Ah, wait. I should have used OPENSSL30_... for it.
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 if(BREW)
 execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
 OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
 execute_process(COMMAND ${BREW} --prefix "openssl"
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?
Wouldn't "openssl" cover all other cases? I don't know how brew works...
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).

If openssl@3.1 isn't installed, brew --prefix openssl is failed.
Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
Triggering java-jars because they seem to be failing on the nightlies due to this reason:
 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

Revision: fc0a16e
Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54
TaskStatus
java-jarsGithub Actions
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:
 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
Jun 29, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Jun 29, 2023
@kou
kou
commented
Jun 29, 2023

Copy link


Copy Markdown

MemberAuthor
gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
@raulcd
raulcd
commented
Jun 29, 2023

Copy link


Copy Markdown

Member
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
Sounds good to me.
@kou
kou
commented
Jun 30, 2023

Copy link


Copy Markdown

MemberAuthor
Created: #36404
I'll merge this.
@kou
kou
merged commit 9d92ed4
into
apache:mainJun 30, 2023
@kou
kou
deleted the

cpp-macos-gandiva-openssl

branch
June 30, 2023 00:41
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Jun 30, 2023
@paleolimbotpaleolimbot
mentioned this pull request
Jul 4, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 5, 2023

Copy link


Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.
There were 5 benchmark results indicating a performance regression:
The full Conbench report has more details.
lriggs pushed a commit
to lriggs/arrow
that referenced
this pull request
Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit
to dremio/arrow
that referenced
this pull request
Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit
to dremio/arrow
that referenced
this pull request
Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS


3 participants

@kou@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-36329: [C++][CI] Use OpenSSL 3 on macOS - #36336
Merged
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Jun 30, 2023
Merged
GH-36329: [C++][CI] Use OpenSSL 3 on macOS#36336
kou merged 2 commits into
apache:mainfrom 
kou:cpp-macos-gandiva-openssl
Conversation
@kou
@koukou
commented
Jun 28, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
GitHub Actions self-hosted runner for macOS has

/usr/local/include/openssl/ provided by OpenSSL 3 (openssl@3). Our include paths have ... -isystem /usr/local/include -isystem /usr/local/opt/openssl@1.1/include .... It means that /usr/local/include/openssl/... is used for #include <openssl/...>.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that $(brew --prefix openssl@3)/include isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking /usr/local/include/openssl by brew unlink openssl@3. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.
@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS
fc0a16e
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@3`). Our
include paths have `... -isystem /usr/local/include -isystem
/usr/local/opt/openssl@1.1/include ...`. It means that
`/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some
problems such as a link error.
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions
self-hosted runner for macOS provides OpenSSL 3 by
/usr/local/include/openssl/. Note that `$(brew --prefix
openssl@3)/include` isn't linked as /usr/local/include/openssl` by
default. So I think that Homebrew GitHub Actions self-hosted runner
for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink
openssl@3`. But there is no reason to use OpenSSL 1.1 for us. So this
PR doesn't use this solution.
@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36329has been automatically assigned in GitHub to PR creator.
@kou
kou
commented
Jun 28, 2023

Copy link


Copy Markdown

MemberAuthor
+1
The "C++ / AMD64 macOS 12 C++" is still failing but it's caused by #36331/#36248 .
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 OUTPUT_STRIP_TRAILING_WHITESPACE)
 if(OPENSSL_BREW_PREFIX)
 set(OPENSSL_ROOT_DIR ${OPENSSL_BREW_PREFIX})
 if(OPENSSL11_BREW_PREFIX)
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be OPENSSL3_BREW_PREFIX?
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Good catch!
Ah, wait. I should have used OPENSSL30_... for it.
pitrou
pitrou
reviewed
Jun 28, 2023
View reviewed changes
Comment threadcpp/cmake_modules/FindOpenSSLAlt.cmake
Outdated

 if(BREW)
 execute_process(COMMAND ${BREW} --prefix "openssl@1.1"
 OUTPUT_VARIABLE OPENSSL11_BREW_PREFIX
 execute_process(COMMAND ${BREW} --prefix "openssl"
@pitroupitrouJun 28, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: why first try "openssl", then "openssl@3.0", then "openssl@1.1"?
Wouldn't "openssl" cover all other cases? I don't know how brew works...
@koukouJun 29, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openssl is the default OpenSSL formula. In general, it refers the latest OpenSSL formula (openssl@3.1 now).

If openssl@3.1 isn't installed, brew --prefix openssl is failed.
Then, this process falls back to openssl@3.0, then openssl@1.1. Because we want to use newer OpenSSL as much as possible.
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
Triggering java-jars because they seem to be failing on the nightlies due to this reason:
 Undefined symbols for architecture arm64:
"_EVP_MD_get_size", referenced from:
gandiva::gdv_hash_using_openssl(long long, void const*, unsigned long, evp_md_st const*, unsigned int, int*) in libgandiva.a(unity_3_cxx.cxx.o)
ld: symbol(s) not found for architecture arm64

@github-actions
github-actionsBot
commented
Jun 28, 2023

Copy link


Copy Markdown

Revision: fc0a16e
Submitted crossbow builds: ursacomputing/crossbow @ actions-b784334a54
TaskStatus
java-jarsGithub Actions
@raulcd
raulcd
commented
Jun 28, 2023

Copy link


Copy Markdown

Member
There seems to be a lot of failures around gandiva on the MacOs java-jars job. I am not sure if they are related:
 The following tests FAILED:
20 - arrow-compute-scalar-type-test (Failed)
38 - arrow-substrait-substrait-test (Failed)
44 - arrow-acero-asof-join-node-test (Failed)
78 - gandiva-internals-test (Failed)
80 - gandiva-filter-test (Failed)
81 - gandiva-projector-test (Failed)
82 - gandiva-projector-build-validation-test (Failed)
83 - gandiva-if-expr-test (Failed)
84 - gandiva-literal-test (Failed)
85 - gandiva-boolean-expr-test (Failed)
86 - gandiva-binary-test (Failed)
87 - gandiva-date-time-test (Failed)
88 - gandiva-to-string-test (Failed)
89 - gandiva-utf8-test (Failed)
90 - gandiva-hash-test (Failed)
91 - gandiva-in-expr-test (Failed)
92 - gandiva-null-validity-test (Failed)
93 - gandiva-decimal-test (Failed)
94 - gandiva-decimal-single-test (Failed)
95 - gandiva-filter-project-test (Failed)
96 - gandiva-projector-test-static (Failed)

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
Jun 29, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Jun 29, 2023
@kou
kou
commented
Jun 29, 2023

Copy link


Copy Markdown

MemberAuthor
gandiva-* tests were crashed without backtrace... So we can't debug this. We may be able debug this by ssh to the host.
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
@raulcd
raulcd
commented
Jun 29, 2023

Copy link


Copy Markdown

Member
Anyway, how about tracking the failures as a separated issue? Because Gandiva tests in "AMD64 macOS 12 C++" aren't failed.
Sounds good to me.
@kou
kou
commented
Jun 30, 2023

Copy link


Copy Markdown

MemberAuthor
Created: #36404
I'll merge this.
@kou
kou
merged commit 9d92ed4
into
apache:mainJun 30, 2023
@kou
kou
deleted the

cpp-macos-gandiva-openssl

branch
June 30, 2023 00:41
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Jun 30, 2023
@paleolimbotpaleolimbot
mentioned this pull request
Jul 4, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 5, 2023

Copy link


Copy Markdown

Conbench analyzed the 6 benchmark runs on commit 9d92ed4d.
There were 5 benchmark results indicating a performance regression:
The full Conbench report has more details.
lriggs pushed a commit
to lriggs/arrow
that referenced
this pull request
Jul 19, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
8c582cd
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
lriggs pushed a commit
to dremio/arrow
that referenced
this pull request
Jul 21, 2023
@kou@lriggs
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336)
6eb8141
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
xxlaykxx added a commit
to dremio/arrow
that referenced
this pull request
Jul 30, 2023
@xxlaykxx@kou
apacheGH-36329: [C++][CI] Use OpenSSL 3 on macOS (apache#36336) (#41)
cfe29f2
### Rationale for this change
GitHub Actions self-hosted runner for macOS has
/usr/local/include/openssl/ provided by OpenSSL 3 (`openssl@ 3`). Our include paths have `... -isystem /usr/local/include -isystem /usr/local/opt/openssl@ 1.1/include ...`. It means that `/usr/local/include/openssl/...` is used for `#include <openssl/...>`.
If we mix OpenSSL 3 headers and OpenSSL 1.1 libraries, we may get some problems such as a link error.
### What changes are included in this PR?
This uses OpenSSL 3 instead of OpenSSL 1.1 because GitHub Actions self-hosted runner for macOS provides OpenSSL 3 by /usr/local/include/openssl/. Note that `$(brew --prefix openssl@ 3)/include` isn't linked as /usr/local/include/openssl` by default. So I think that Homebrew GitHub Actions self-hosted runner for macOS does it explicitly.
Other solution: Unlinking `/usr/local/include/openssl` by `brew unlink openssl@ 3`. But there is no reason to use OpenSSL 1.1 for us. So this PR doesn't use this solution.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: apache#36329
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou left review comments

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++][CI] OpenSSL link error in Gandiva on macOS


3 participants

@kou@raulcd@pitrou