ARROW-16584: [Java] Java JNI with S3 support - #13157

Merged
kou merged 12 commits into
apache:masterfrom
REASY:feature/java-with-s3
Jul 29, 2022
Merged

ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom
REASY:feature/java-with-s3

Conversation

@REASY

@REASYREASY commented May 14, 2022•
edited by kou
Loading

Copy link
Copy Markdown
Contributor

macOS development target is changed to 10.13 from 10.11.
See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w

@github-actions

github-actionsBot commented May 14, 2022

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@REASYREASY mentioned this pull request May 14, 2022
Closed
@kou

kou commented May 14, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@github-actions

github-actionsBot commented May 14, 2022

Copy link
Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf

Submitted crossbow builds: ursacomputing/crossbow @ actions-2091

TaskStatus
java-jarsGithub Actions

REASY
REASY commented May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile Outdated
kou
kou reviewed May 15, 2022
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?

Could you also follow #13157 (comment) ?

Comment threadci/docker/linux-apt-jni.dockerfile Outdated
Comment threadci/docker/linux-apt-jni.dockerfile Outdated
Comment threadci/scripts/java_jni_manylinux_build.sh Outdated
@REASYREASY changed the title feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions

github-actionsBot commented May 16, 2022

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584

@github-actions

github-actionsBot commented May 16, 2022

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@REASY
REASY marked this pull request as ready for review May 17, 2022 01:36
@REASY
REASY requested a review from kouMay 18, 2022 08:03
@kou

kou commented May 18, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@github-actions

github-actionsBot commented May 18, 2022

Copy link
Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24

Submitted crossbow builds: ursacomputing/crossbow @ actions-2110

TaskStatus
java-jarsGithub Actions

@REASY

REASY commented May 18, 2022

Copy link
Copy Markdown
ContributorAuthor

@kou do you think this can be backported to 8.x branch?
Could you point to release cycle, maybe 9.x is coming soo though...

@kou

kou commented May 18, 2022

Copy link
Copy Markdown
Member

It seems that we need to link AWS SDK C++ statically:

https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639

Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?

diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \

@kou

kou commented May 18, 2022

Copy link
Copy Markdown
Member

@kou do you think this can be backported to 8.x branch?

No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.

Could you point to release cycle, maybe 9.x is coming soo though...

Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.

FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt
We may release more frequently in future.

@REASY
REASYforce-pushed the feature/java-with-s3 branch from df01c4b to ad62c6fCompareMay 19, 2022 00:50
@raulcd

raulcd commented May 20, 2022

Copy link
Copy Markdown
Member

@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116
We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.
Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense

@kou

kou commented May 21, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@github-actions

github-actionsBot commented May 21, 2022

Copy link
Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043

Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91

TaskStatus
java-jarsGithub Actions

@kou

kou commented May 21, 2022

Copy link
Copy Markdown
Member

@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)

I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)

@kou

kou commented May 22, 2022

Copy link
Copy Markdown
Member

Could you rebase on master? It may resolve the java-jars build failure.

@REASY
REASYforce-pushed the feature/java-with-s3 branch from ad62c6f to c2733b0CompareMay 22, 2022 13:44
@REASY

REASY commented May 22, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403

Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.

@kou

kou commented May 22, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@github-actions

github-actionsBot commented May 22, 2022

Copy link
Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311

Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc

TaskStatus
java-jarsGithub Actions

@kou

kou commented May 23, 2022

Copy link
Copy Markdown
Member

Thanks for the warning information! Then we should test with multiple JDK versions.

@raulcd

raulcd commented May 24, 2022

Copy link
Copy Markdown
Member

I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!

@REASY

REASY commented May 28, 2022

Copy link
Copy Markdown
ContributorAuthor

@kou what are the next steps to make this PR ready?

@kou

kou commented May 28, 2022

Copy link
Copy Markdown
Member

We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.

@REASY

REASY commented Jul 26, 2022

Copy link
Copy Markdown
ContributorAuthor

@kou the error is the same as in previous java-jars:

Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed the feature/java-with-s3 branch from 53a8006 to d482759CompareJuly 26, 2022 03:45
@REASY

REASY commented Jul 27, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:

Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:

ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:

ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?

brew's version for aws-sdk-cpp is 1.9.300

@kou

kou commented Jul 28, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@github-actions

github-actionsBot commented Jul 28, 2022

Copy link
Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f

Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f

TaskStatus
java-jarsGithub Actions

@kou

kou commented Jul 28, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@github-actions

github-actionsBot commented Jul 28, 2022

Copy link
Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd

Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62

TaskStatus
java-jarsGithub Actions

REASYand others added 11 commits July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed the feature/java-with-s3 branch from f3c2263 to 3c7255aCompareJuly 28, 2022 12:04
@REASY

REASY commented Jul 28, 2022

Copy link
Copy Markdown
ContributorAuthor

Revision: 1ac0510

Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62

Task Status
java-jars Github Actions

This should be fixed by #13712

@kou

kou commented Jul 29, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

@github-actions

github-actionsBot commented Jul 29, 2022

Copy link
Copy Markdown

Revision: 3c7255a

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96

TaskStatus
java-jarsGithub Actions

kou
kou approved these changes Jul 29, 2022
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Thanks!

@kou
kou merged commit 7fe71f5 into apache:masterJul 29, 2022
@REASY

REASY commented Jul 29, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Thanks, @kou!

Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see

 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?

@kou

kou commented Jul 29, 2022

Copy link
Copy Markdown
Member

Sure.

Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.

Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.

@ursabot

ursabot commented Jul 29, 2022

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q
Buildkite builds:
[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2
[Finished] 7fe71f5c test-mac-arm
[Finished] 7fe71f5c ursa-i9-9960x
[Finished] 7fe71f5c ursa-thinkcentre-m75q
[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2
[Failed] fa7e7a3f test-mac-arm
[Finished] fa7e7a3f ursa-i9-9960x
[Finished] fa7e7a3f ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@koukou mentioned this pull request Aug 19, 2022
Merged
@asfimportasfimport mentioned this pull request Nov 26, 2024
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-16584: [Java] Java JNI with S3 support - #13157
Merged
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Jul 29, 2022
Merged
ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Conversation
@REASY
@REASYREASY
commented
May 14, 2022•

edited by kou

Loading

Copy link


Copy Markdown

Contributor
macOS development target is changed to 10.13 from 10.11.

See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

See also:
@REASYREASY
mentioned this pull request
May 14, 2022

Closed

@kou
kou
commented
May 14, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf
Submitted crossbow builds: ursacomputing/crossbow @ actions-2091
TaskStatus
java-jarsGithub Actions
REASY
REASY
commented
May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

kou
kou
reviewed
May 15, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?
Could you also follow #13157 (comment) ?
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/scripts/java_jni_manylinux_build.sh
Outdated

@REASYREASY
changed the title
feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
@REASY
REASY
marked this pull request as ready for review
May 17, 2022 01:36
@REASY
REASY
requested a review
from kouMay 18, 2022 08:03
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 18, 2022

Copy link


Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24
Submitted crossbow builds: ursacomputing/crossbow @ actions-2110
TaskStatus
java-jarsGithub Actions
@REASY
REASY
commented
May 18, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou do you think this can be backported to 8.x branch?

Could you point to release cycle, maybe 9.x is coming soo though...
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
It seems that we need to link AWS SDK C++ statically:
https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639
Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?
diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@kou do you think this can be backported to 8.x branch?
No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.
Could you point to release cycle, maybe 9.x is coming soo though...
Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.
FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt

We may release more frequently in future.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
df01c4b to
ad62c6fCompareMay 19, 2022 00:50
@raulcd
raulcd
commented
May 20, 2022

Copy link


Copy Markdown

Member
@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116

We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.

Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 21, 2022

Copy link


Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043
Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)
I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
Could you rebase on master? It may resolve the java-jars build failure.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
ad62c6f to
c2733b0CompareMay 22, 2022 13:44
@REASY
REASY
commented
May 22, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403
Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 22, 2022

Copy link


Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311
Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 23, 2022

Copy link


Copy Markdown

Member
Thanks for the warning information! Then we should test with multiple JDK versions.
@raulcd
raulcd
commented
May 24, 2022

Copy link


Copy Markdown

Member
I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!
@REASY
REASY
commented
May 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou what are the next steps to make this PR ready?
@kou
kou
commented
May 28, 2022

Copy link


Copy Markdown

Member
We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.
@REASY
REASY
commented
Jul 26, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou the error is the same as in previous java-jars:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
53a8006 to
d482759CompareJuly 26, 2022 03:45
@REASY
REASY
commented
Jul 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?
brew's version for aws-sdk-cpp is 1.9.300
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f
Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
TaskStatus
java-jarsGithub Actions
REASYand others
added 11 commits
July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
f3c2263 to
3c7255aCompareJuly 28, 2022 12:04
@REASY
REASY
commented
Jul 28, 2022

Copy link


Copy Markdown

ContributorAuthor
Revision: 1ac0510
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
Task	Status

java-jars	Github Actions
This should be fixed by #13712
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 29, 2022

Copy link


Copy Markdown

Revision: 3c7255a
Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96
TaskStatus
java-jarsGithub Actions
kou
kou
approved these changes
Jul 29, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Thanks!
@kou
kou
merged commit 7fe71f5
into
apache:masterJul 29, 2022
@REASY
REASY
commented
Jul 29, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Thanks, @kou!
Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see
 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
Sure.
Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.
Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.
@ursabot
ursabot
commented
Jul 29, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q

Buildkite builds:

[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2

[Finished] 7fe71f5c test-mac-arm

[Finished] 7fe71f5c ursa-i9-9960x

[Finished] 7fe71f5c ursa-thinkcentre-m75q

[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2

[Failed] fa7e7a3f test-mac-arm

[Finished] fa7e7a3f ursa-i9-9960x

[Finished] fa7e7a3f ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@koukou
mentioned this pull request
Aug 19, 2022
 Merged

@asfimportasfimport
mentioned this pull request
Nov 26, 2024

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-16584: [Java] Java JNI with S3 support - #13157
Merged
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Jul 29, 2022
Merged
ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Conversation
@REASY
@REASYREASY
commented
May 14, 2022•

edited by kou

Loading

Copy link


Copy Markdown

Contributor
macOS development target is changed to 10.13 from 10.11.

See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

See also:
@REASYREASY
mentioned this pull request
May 14, 2022

Closed

@kou
kou
commented
May 14, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf
Submitted crossbow builds: ursacomputing/crossbow @ actions-2091
TaskStatus
java-jarsGithub Actions
REASY
REASY
commented
May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

kou
kou
reviewed
May 15, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?
Could you also follow #13157 (comment) ?
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/scripts/java_jni_manylinux_build.sh
Outdated

@REASYREASY
changed the title
feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
@REASY
REASY
marked this pull request as ready for review
May 17, 2022 01:36
@REASY
REASY
requested a review
from kouMay 18, 2022 08:03
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 18, 2022

Copy link


Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24
Submitted crossbow builds: ursacomputing/crossbow @ actions-2110
TaskStatus
java-jarsGithub Actions
@REASY
REASY
commented
May 18, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou do you think this can be backported to 8.x branch?

Could you point to release cycle, maybe 9.x is coming soo though...
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
It seems that we need to link AWS SDK C++ statically:
https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639
Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?
diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@kou do you think this can be backported to 8.x branch?
No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.
Could you point to release cycle, maybe 9.x is coming soo though...
Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.
FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt

We may release more frequently in future.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
df01c4b to
ad62c6fCompareMay 19, 2022 00:50
@raulcd
raulcd
commented
May 20, 2022

Copy link


Copy Markdown

Member
@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116

We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.

Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 21, 2022

Copy link


Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043
Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)
I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
Could you rebase on master? It may resolve the java-jars build failure.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
ad62c6f to
c2733b0CompareMay 22, 2022 13:44
@REASY
REASY
commented
May 22, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403
Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 22, 2022

Copy link


Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311
Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 23, 2022

Copy link


Copy Markdown

Member
Thanks for the warning information! Then we should test with multiple JDK versions.
@raulcd
raulcd
commented
May 24, 2022

Copy link


Copy Markdown

Member
I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!
@REASY
REASY
commented
May 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou what are the next steps to make this PR ready?
@kou
kou
commented
May 28, 2022

Copy link


Copy Markdown

Member
We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.
@REASY
REASY
commented
Jul 26, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou the error is the same as in previous java-jars:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
53a8006 to
d482759CompareJuly 26, 2022 03:45
@REASY
REASY
commented
Jul 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?
brew's version for aws-sdk-cpp is 1.9.300
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f
Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
TaskStatus
java-jarsGithub Actions
REASYand others
added 11 commits
July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
f3c2263 to
3c7255aCompareJuly 28, 2022 12:04
@REASY
REASY
commented
Jul 28, 2022

Copy link


Copy Markdown

ContributorAuthor
Revision: 1ac0510
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
Task	Status

java-jars	Github Actions
This should be fixed by #13712
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 29, 2022

Copy link


Copy Markdown

Revision: 3c7255a
Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96
TaskStatus
java-jarsGithub Actions
kou
kou
approved these changes
Jul 29, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Thanks!
@kou
kou
merged commit 7fe71f5
into
apache:masterJul 29, 2022
@REASY
REASY
commented
Jul 29, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Thanks, @kou!
Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see
 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
Sure.
Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.
Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.
@ursabot
ursabot
commented
Jul 29, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q

Buildkite builds:

[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2

[Finished] 7fe71f5c test-mac-arm

[Finished] 7fe71f5c ursa-i9-9960x

[Finished] 7fe71f5c ursa-thinkcentre-m75q

[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2

[Failed] fa7e7a3f test-mac-arm

[Finished] fa7e7a3f ursa-i9-9960x

[Finished] fa7e7a3f ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@koukou
mentioned this pull request
Aug 19, 2022
 Merged

@asfimportasfimport
mentioned this pull request
Nov 26, 2024

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-16584: [Java] Java JNI with S3 support - #13157
Merged
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Jul 29, 2022
Merged
ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Conversation
@REASY
@REASYREASY
commented
May 14, 2022•

edited by kou

Loading

Copy link


Copy Markdown

Contributor
macOS development target is changed to 10.13 from 10.11.

See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

See also:
@REASYREASY
mentioned this pull request
May 14, 2022

Closed

@kou
kou
commented
May 14, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf
Submitted crossbow builds: ursacomputing/crossbow @ actions-2091
TaskStatus
java-jarsGithub Actions
REASY
REASY
commented
May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

kou
kou
reviewed
May 15, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?
Could you also follow #13157 (comment) ?
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/scripts/java_jni_manylinux_build.sh
Outdated

@REASYREASY
changed the title
feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
@REASY
REASY
marked this pull request as ready for review
May 17, 2022 01:36
@REASY
REASY
requested a review
from kouMay 18, 2022 08:03
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 18, 2022

Copy link


Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24
Submitted crossbow builds: ursacomputing/crossbow @ actions-2110
TaskStatus
java-jarsGithub Actions
@REASY
REASY
commented
May 18, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou do you think this can be backported to 8.x branch?

Could you point to release cycle, maybe 9.x is coming soo though...
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
It seems that we need to link AWS SDK C++ statically:
https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639
Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?
diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@kou do you think this can be backported to 8.x branch?
No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.
Could you point to release cycle, maybe 9.x is coming soo though...
Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.
FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt

We may release more frequently in future.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
df01c4b to
ad62c6fCompareMay 19, 2022 00:50
@raulcd
raulcd
commented
May 20, 2022

Copy link


Copy Markdown

Member
@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116

We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.

Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 21, 2022

Copy link


Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043
Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)
I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
Could you rebase on master? It may resolve the java-jars build failure.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
ad62c6f to
c2733b0CompareMay 22, 2022 13:44
@REASY
REASY
commented
May 22, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403
Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 22, 2022

Copy link


Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311
Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 23, 2022

Copy link


Copy Markdown

Member
Thanks for the warning information! Then we should test with multiple JDK versions.
@raulcd
raulcd
commented
May 24, 2022

Copy link


Copy Markdown

Member
I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!
@REASY
REASY
commented
May 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou what are the next steps to make this PR ready?
@kou
kou
commented
May 28, 2022

Copy link


Copy Markdown

Member
We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.
@REASY
REASY
commented
Jul 26, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou the error is the same as in previous java-jars:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
53a8006 to
d482759CompareJuly 26, 2022 03:45
@REASY
REASY
commented
Jul 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?
brew's version for aws-sdk-cpp is 1.9.300
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f
Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
TaskStatus
java-jarsGithub Actions
REASYand others
added 11 commits
July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
f3c2263 to
3c7255aCompareJuly 28, 2022 12:04
@REASY
REASY
commented
Jul 28, 2022

Copy link


Copy Markdown

ContributorAuthor
Revision: 1ac0510
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
Task	Status

java-jars	Github Actions
This should be fixed by #13712
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 29, 2022

Copy link


Copy Markdown

Revision: 3c7255a
Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96
TaskStatus
java-jarsGithub Actions
kou
kou
approved these changes
Jul 29, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Thanks!
@kou
kou
merged commit 7fe71f5
into
apache:masterJul 29, 2022
@REASY
REASY
commented
Jul 29, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Thanks, @kou!
Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see
 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
Sure.
Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.
Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.
@ursabot
ursabot
commented
Jul 29, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q

Buildkite builds:

[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2

[Finished] 7fe71f5c test-mac-arm

[Finished] 7fe71f5c ursa-i9-9960x

[Finished] 7fe71f5c ursa-thinkcentre-m75q

[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2

[Failed] fa7e7a3f test-mac-arm

[Finished] fa7e7a3f ursa-i9-9960x

[Finished] fa7e7a3f ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@koukou
mentioned this pull request
Aug 19, 2022
 Merged

@asfimportasfimport
mentioned this pull request
Nov 26, 2024

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-16584: [Java] Java JNI with S3 support - #13157
Merged
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Jul 29, 2022
Merged
ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Conversation
@REASY
@REASYREASY
commented
May 14, 2022•

edited by kou

Loading

Copy link


Copy Markdown

Contributor
macOS development target is changed to 10.13 from 10.11.

See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

See also:
@REASYREASY
mentioned this pull request
May 14, 2022

Closed

@kou
kou
commented
May 14, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf
Submitted crossbow builds: ursacomputing/crossbow @ actions-2091
TaskStatus
java-jarsGithub Actions
REASY
REASY
commented
May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

kou
kou
reviewed
May 15, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?
Could you also follow #13157 (comment) ?
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/scripts/java_jni_manylinux_build.sh
Outdated

@REASYREASY
changed the title
feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
@REASY
REASY
marked this pull request as ready for review
May 17, 2022 01:36
@REASY
REASY
requested a review
from kouMay 18, 2022 08:03
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 18, 2022

Copy link


Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24
Submitted crossbow builds: ursacomputing/crossbow @ actions-2110
TaskStatus
java-jarsGithub Actions
@REASY
REASY
commented
May 18, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou do you think this can be backported to 8.x branch?

Could you point to release cycle, maybe 9.x is coming soo though...
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
It seems that we need to link AWS SDK C++ statically:
https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639
Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?
diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@kou do you think this can be backported to 8.x branch?
No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.
Could you point to release cycle, maybe 9.x is coming soo though...
Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.
FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt

We may release more frequently in future.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
df01c4b to
ad62c6fCompareMay 19, 2022 00:50
@raulcd
raulcd
commented
May 20, 2022

Copy link


Copy Markdown

Member
@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116

We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.

Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 21, 2022

Copy link


Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043
Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)
I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
Could you rebase on master? It may resolve the java-jars build failure.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
ad62c6f to
c2733b0CompareMay 22, 2022 13:44
@REASY
REASY
commented
May 22, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403
Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 22, 2022

Copy link


Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311
Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 23, 2022

Copy link


Copy Markdown

Member
Thanks for the warning information! Then we should test with multiple JDK versions.
@raulcd
raulcd
commented
May 24, 2022

Copy link


Copy Markdown

Member
I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!
@REASY
REASY
commented
May 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou what are the next steps to make this PR ready?
@kou
kou
commented
May 28, 2022

Copy link


Copy Markdown

Member
We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.
@REASY
REASY
commented
Jul 26, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou the error is the same as in previous java-jars:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
53a8006 to
d482759CompareJuly 26, 2022 03:45
@REASY
REASY
commented
Jul 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?
brew's version for aws-sdk-cpp is 1.9.300
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f
Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
TaskStatus
java-jarsGithub Actions
REASYand others
added 11 commits
July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
f3c2263 to
3c7255aCompareJuly 28, 2022 12:04
@REASY
REASY
commented
Jul 28, 2022

Copy link


Copy Markdown

ContributorAuthor
Revision: 1ac0510
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
Task	Status

java-jars	Github Actions
This should be fixed by #13712
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 29, 2022

Copy link


Copy Markdown

Revision: 3c7255a
Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96
TaskStatus
java-jarsGithub Actions
kou
kou
approved these changes
Jul 29, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Thanks!
@kou
kou
merged commit 7fe71f5
into
apache:masterJul 29, 2022
@REASY
REASY
commented
Jul 29, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Thanks, @kou!
Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see
 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
Sure.
Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.
Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.
@ursabot
ursabot
commented
Jul 29, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q

Buildkite builds:

[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2

[Finished] 7fe71f5c test-mac-arm

[Finished] 7fe71f5c ursa-i9-9960x

[Finished] 7fe71f5c ursa-thinkcentre-m75q

[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2

[Failed] fa7e7a3f test-mac-arm

[Finished] fa7e7a3f ursa-i9-9960x

[Finished] fa7e7a3f ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@koukou
mentioned this pull request
Aug 19, 2022
 Merged

@asfimportasfimport
mentioned this pull request
Nov 26, 2024

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-16584: [Java] Java JNI with S3 support - #13157
Merged
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Jul 29, 2022
Merged
ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Conversation
@REASY
@REASYREASY
commented
May 14, 2022•

edited by kou

Loading

Copy link


Copy Markdown

Contributor
macOS development target is changed to 10.13 from 10.11.

See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

See also:
@REASYREASY
mentioned this pull request
May 14, 2022

Closed

@kou
kou
commented
May 14, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf
Submitted crossbow builds: ursacomputing/crossbow @ actions-2091
TaskStatus
java-jarsGithub Actions
REASY
REASY
commented
May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

kou
kou
reviewed
May 15, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?
Could you also follow #13157 (comment) ?
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/scripts/java_jni_manylinux_build.sh
Outdated

@REASYREASY
changed the title
feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
@REASY
REASY
marked this pull request as ready for review
May 17, 2022 01:36
@REASY
REASY
requested a review
from kouMay 18, 2022 08:03
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 18, 2022

Copy link


Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24
Submitted crossbow builds: ursacomputing/crossbow @ actions-2110
TaskStatus
java-jarsGithub Actions
@REASY
REASY
commented
May 18, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou do you think this can be backported to 8.x branch?

Could you point to release cycle, maybe 9.x is coming soo though...
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
It seems that we need to link AWS SDK C++ statically:
https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639
Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?
diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@kou do you think this can be backported to 8.x branch?
No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.
Could you point to release cycle, maybe 9.x is coming soo though...
Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.
FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt

We may release more frequently in future.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
df01c4b to
ad62c6fCompareMay 19, 2022 00:50
@raulcd
raulcd
commented
May 20, 2022

Copy link


Copy Markdown

Member
@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116

We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.

Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 21, 2022

Copy link


Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043
Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)
I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
Could you rebase on master? It may resolve the java-jars build failure.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
ad62c6f to
c2733b0CompareMay 22, 2022 13:44
@REASY
REASY
commented
May 22, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403
Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 22, 2022

Copy link


Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311
Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 23, 2022

Copy link


Copy Markdown

Member
Thanks for the warning information! Then we should test with multiple JDK versions.
@raulcd
raulcd
commented
May 24, 2022

Copy link


Copy Markdown

Member
I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!
@REASY
REASY
commented
May 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou what are the next steps to make this PR ready?
@kou
kou
commented
May 28, 2022

Copy link


Copy Markdown

Member
We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.
@REASY
REASY
commented
Jul 26, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou the error is the same as in previous java-jars:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
53a8006 to
d482759CompareJuly 26, 2022 03:45
@REASY
REASY
commented
Jul 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?
brew's version for aws-sdk-cpp is 1.9.300
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f
Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
TaskStatus
java-jarsGithub Actions
REASYand others
added 11 commits
July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
f3c2263 to
3c7255aCompareJuly 28, 2022 12:04
@REASY
REASY
commented
Jul 28, 2022

Copy link


Copy Markdown

ContributorAuthor
Revision: 1ac0510
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
Task	Status

java-jars	Github Actions
This should be fixed by #13712
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 29, 2022

Copy link


Copy Markdown

Revision: 3c7255a
Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96
TaskStatus
java-jarsGithub Actions
kou
kou
approved these changes
Jul 29, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Thanks!
@kou
kou
merged commit 7fe71f5
into
apache:masterJul 29, 2022
@REASY
REASY
commented
Jul 29, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Thanks, @kou!
Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see
 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
Sure.
Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.
Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.
@ursabot
ursabot
commented
Jul 29, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q

Buildkite builds:

[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2

[Finished] 7fe71f5c test-mac-arm

[Finished] 7fe71f5c ursa-i9-9960x

[Finished] 7fe71f5c ursa-thinkcentre-m75q

[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2

[Failed] fa7e7a3f test-mac-arm

[Finished] fa7e7a3f ursa-i9-9960x

[Finished] fa7e7a3f ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@koukou
mentioned this pull request
Aug 19, 2022
 Merged

@asfimportasfimport
mentioned this pull request
Nov 26, 2024

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-16584: [Java] Java JNI with S3 support - #13157
Merged
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Jul 29, 2022
Merged
ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Conversation
@REASY
@REASYREASY
commented
May 14, 2022•

edited by kou

Loading

Copy link


Copy Markdown

Contributor
macOS development target is changed to 10.13 from 10.11.

See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

See also:
@REASYREASY
mentioned this pull request
May 14, 2022

Closed

@kou
kou
commented
May 14, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf
Submitted crossbow builds: ursacomputing/crossbow @ actions-2091
TaskStatus
java-jarsGithub Actions
REASY
REASY
commented
May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

kou
kou
reviewed
May 15, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?
Could you also follow #13157 (comment) ?
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/scripts/java_jni_manylinux_build.sh
Outdated

@REASYREASY
changed the title
feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
@REASY
REASY
marked this pull request as ready for review
May 17, 2022 01:36
@REASY
REASY
requested a review
from kouMay 18, 2022 08:03
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 18, 2022

Copy link


Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24
Submitted crossbow builds: ursacomputing/crossbow @ actions-2110
TaskStatus
java-jarsGithub Actions
@REASY
REASY
commented
May 18, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou do you think this can be backported to 8.x branch?

Could you point to release cycle, maybe 9.x is coming soo though...
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
It seems that we need to link AWS SDK C++ statically:
https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639
Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?
diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@kou do you think this can be backported to 8.x branch?
No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.
Could you point to release cycle, maybe 9.x is coming soo though...
Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.
FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt

We may release more frequently in future.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
df01c4b to
ad62c6fCompareMay 19, 2022 00:50
@raulcd
raulcd
commented
May 20, 2022

Copy link


Copy Markdown

Member
@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116

We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.

Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 21, 2022

Copy link


Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043
Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)
I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
Could you rebase on master? It may resolve the java-jars build failure.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
ad62c6f to
c2733b0CompareMay 22, 2022 13:44
@REASY
REASY
commented
May 22, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403
Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 22, 2022

Copy link


Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311
Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 23, 2022

Copy link


Copy Markdown

Member
Thanks for the warning information! Then we should test with multiple JDK versions.
@raulcd
raulcd
commented
May 24, 2022

Copy link


Copy Markdown

Member
I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!
@REASY
REASY
commented
May 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou what are the next steps to make this PR ready?
@kou
kou
commented
May 28, 2022

Copy link


Copy Markdown

Member
We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.
@REASY
REASY
commented
Jul 26, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou the error is the same as in previous java-jars:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
53a8006 to
d482759CompareJuly 26, 2022 03:45
@REASY
REASY
commented
Jul 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?
brew's version for aws-sdk-cpp is 1.9.300
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f
Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
TaskStatus
java-jarsGithub Actions
REASYand others
added 11 commits
July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
f3c2263 to
3c7255aCompareJuly 28, 2022 12:04
@REASY
REASY
commented
Jul 28, 2022

Copy link


Copy Markdown

ContributorAuthor
Revision: 1ac0510
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
Task	Status

java-jars	Github Actions
This should be fixed by #13712
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 29, 2022

Copy link


Copy Markdown

Revision: 3c7255a
Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96
TaskStatus
java-jarsGithub Actions
kou
kou
approved these changes
Jul 29, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Thanks!
@kou
kou
merged commit 7fe71f5
into
apache:masterJul 29, 2022
@REASY
REASY
commented
Jul 29, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Thanks, @kou!
Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see
 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
Sure.
Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.
Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.
@ursabot
ursabot
commented
Jul 29, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q

Buildkite builds:

[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2

[Finished] 7fe71f5c test-mac-arm

[Finished] 7fe71f5c ursa-i9-9960x

[Finished] 7fe71f5c ursa-thinkcentre-m75q

[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2

[Failed] fa7e7a3f test-mac-arm

[Finished] fa7e7a3f ursa-i9-9960x

[Finished] fa7e7a3f ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@koukou
mentioned this pull request
Aug 19, 2022
 Merged

@asfimportasfimport
mentioned this pull request
Nov 26, 2024

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-16584: [Java] Java JNI with S3 support - #13157
Merged
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Jul 29, 2022
Merged
ARROW-16584: [Java] Java JNI with S3 support#13157
kou merged 12 commits into
apache:masterfrom 
REASY:feature/java-with-s3
Conversation
@REASY
@REASYREASY
commented
May 14, 2022•

edited by kou

Loading

Copy link


Copy Markdown

Contributor
macOS development target is changed to 10.13 from 10.11.

See also the discussion on mailing list:
https://lists.apache.org/thread/pjgjrl716gvqzql586cnnoxb38nb0j5w
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

See also:
@REASYREASY
mentioned this pull request
May 14, 2022

Closed

@kou
kou
commented
May 14, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 14, 2022

Copy link


Copy Markdown

Revision: 777e1ef69ae50e071251bf742a6109dbb3b250bf
Submitted crossbow builds: ursacomputing/crossbow @ actions-2091
TaskStatus
java-jarsGithub Actions
REASY
REASY
commented
May 15, 2022
View reviewed changes
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

kou
kou
reviewed
May 15, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update ci/scripts/java_jni_macos_build.sh?
Could you also follow #13157 (comment) ?
Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/docker/linux-apt-jni.dockerfile
Outdated

Comment threadci/scripts/java_jni_manylinux_build.sh
Outdated

@REASYREASY
changed the title
feat: Java JNI with S3 supportARROW-16584: [Java] Java JNI with S3 supportMay 16, 2022
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-16584
@github-actions
github-actionsBot
commented
May 16, 2022

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
@REASY
REASY
marked this pull request as ready for review
May 17, 2022 01:36
@REASY
REASY
requested a review
from kouMay 18, 2022 08:03
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 18, 2022

Copy link


Copy Markdown

Revision: df01c4bddd35f86f1cc487d7d851d8278b732a24
Submitted crossbow builds: ursacomputing/crossbow @ actions-2110
TaskStatus
java-jarsGithub Actions
@REASY
REASY
commented
May 18, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou do you think this can be backported to 8.x branch?

Could you point to release cycle, maybe 9.x is coming soo though...
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
It seems that we need to link AWS SDK C++ statically:
https://github.com/ursacomputing/crossbow/runs/6485814040?check_suite_focus=true#step:7:4639
Error: Unexpected shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`

Could you add the following build option?
diff --git a/ci/scripts/java_jni_macos_build.sh b/ci/scripts/java_jni_macos_build.sh
index e9572c334b..71a1730fce 100755
--- a/ci/scripts/java_jni_macos_build.sh+++ b/ci/scripts/java_jni_macos_build.sh@@ -76,6 +76,7 @@ cmake \
-DARROW_THRIFT_USE_SHARED=OFF \
-DARROW_UTF8PROC_USE_SHARED=OFF \
-DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLE \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=${build_dir} \
@kou
kou
commented
May 18, 2022

Copy link


Copy Markdown

Member
@kou do you think this can be backported to 8.x branch?
No. Generally, we backport only bug fixes. But we can discuss this on the dev@arrow.apache.org mailing list.
Could you point to release cycle, maybe 9.x is coming soo though...
Generally, we release a new version each 3-4 months. 8.0.0 was released in 2022-05. So 9.0.0 will be released in 2022-08/09.
FYI: We discussed release cycle recently: https://lists.apache.org/thread/g6mqpyq2hc11xbgrq2pf653njzy53plt

We may release more frequently in future.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
df01c4b to
ad62c6fCompareMay 19, 2022 00:50
@raulcd
raulcd
commented
May 20, 2022

Copy link


Copy Markdown

Member
@kou@kszucs as a follow up from this issue, I can see we generate the jars on our nightly tests via the java-jars task but we don't seem to test them https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml#L109-L116

We also seem to build them on java-jni-manylinux-2014 but not test the ones built either and only with JDK 1.8.

Am I missing us testing them somewhere else? Should we add new nightly jobs that builds on several JDKs and runs tests over the generated binaries? I can work on that if it would make sense
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 21, 2022

Copy link


Copy Markdown

Revision: ad62c6f561d846838dd868dc2fb992706005e043
Submitted crossbow builds: ursacomputing/crossbow @ actions-f6bd02cb91
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 21, 2022

Copy link


Copy Markdown

Member
@raulcd Right. We don't test the built jars for now. We should test them. We can add extra (GitHub Actions) jobs to https://github.com/apache/arrow/blob/master/dev/tasks/java-jars/github.yml to test the built jars on Linux and macOS. (We don't need to add extra Crossbow jobs.)
I don't know much about JDK versions. Can we assume that the built jars with JDK 1.8 can work with all JDK >= 1.8? Or should we build jars for JDK X with JDK X? If we can assume that the built jars with JDK 1.8 can work with all JDK >= 1.8, we can just build jars only with JDK 1.8. (It may be better that we test the built jars with multiple JDK versions.)
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
Could you rebase on master? It may resolve the java-jars build failure.
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
ad62c6f to
c2733b0CompareMay 22, 2022 13:44
@REASY
REASY
commented
May 22, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou I think it makes sense to test separately for Java 8 and Java 11. Java 11 has new features, so the bytecode could look different. I'm getting the following warning on Java 11 when running parquet reader locally:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.arrow.memory.util.MemoryUtil (file:/home/%USER%/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.arrow.memory.util.MemoryUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If arrow java supports Java 17, it has to be tested separately as well because of the following change in JDK 17: https://openjdk.java.net/jeps/403
Strongly encapsulate all internal elements of the JDK, except for critical internal APIs such as sun.misc.Unsafe. It will no longer be possible to relax the strong encapsulation of internal elements via a single command-line option, as was possible in JDK 9 through JDK 16.
@kou
kou
commented
May 22, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
May 22, 2022

Copy link


Copy Markdown

Revision: c2733b067fe04801e783acc8ec37349dffa00311
Submitted crossbow builds: ursacomputing/crossbow @ actions-04af25fddc
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
May 23, 2022

Copy link


Copy Markdown

Member
Thanks for the warning information! Then we should test with multiple JDK versions.
@raulcd
raulcd
commented
May 24, 2022

Copy link


Copy Markdown

Member
I have created https://issues.apache.org/jira/browse/ARROW-16640 to track what we discussed about testing the jars and extend to other JDKs. Thanks!
@REASY
REASY
commented
May 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou what are the next steps to make this PR ready?
@kou
kou
commented
May 28, 2022

Copy link


Copy Markdown

Member
We need to fix the https://github.com/ursacomputing/crossbow/runs/6547084325?check_suite_focus=true failure.
@REASY
REASY
commented
Jul 26, 2022

Copy link


Copy Markdown

ContributorAuthor
@kou the error is the same as in previous java-jars:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
53a8006 to
d482759CompareJuly 26, 2022 03:45
@REASY
REASY
commented
Jul 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
I've tried to add BUILD_SHARED_LIBS (related to aws/aws-sdk-cpp#1334 (comment))
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 5d1da18b7..f8679b04f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4552,6 +4552,7 @@ macro(build_awssdk)
set(AWSSDK_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/awssdk_ep-install")
set(AWSSDK_INCLUDE_DIR "${AWSSDK_PREFIX}/include")
set(AWSSDK_LIB_DIR "lib")
+ option(BUILD_SHARED_LIBS "Build shared libraries" ON)
if(WIN32)
# On Windows, need to match build types
@@ -4781,6 +4782,7 @@ if(ARROW_S3)
set_target_properties(AWS::aws-c-common
PROPERTIES INTERFACE_LINK_LIBRARIES
"-pthread;pthread;-framework CoreFoundation")
+ set(BUILD_SHARED_LIBS "ON")
endif()
endif()

but getting the same error:
Undefined symbols for architecture x86_64:
"Aws::S3::S3Client::GetObjectAttributes(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponse(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesAsync(Aws::S3::Model::GetObjectAttributesRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::GetObjectAttributesRequest const&, Aws::Utils::Outcome<Aws::S3::Model::GetObjectAttributesResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::GetObjectAttributesCallable(Aws::S3::Model::GetObjectAttributesRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseAsync(Aws::S3::Model::WriteGetObjectResponseRequest const&, std::__1::function<void (Aws::S3::S3Client const*, Aws::S3::Model::WriteGetObjectResponseRequest const&, Aws::Utils::Outcome<Aws::NoResult, Aws::S3::S3Error> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::__1::shared_ptr<Aws::Client::AsyncCallerContext const> const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
"Aws::S3::S3Client::WriteGetObjectResponseCallable(Aws::S3::Model::WriteGetObjectResponseRequest const&) const", referenced from:
vtable for arrow::fs::(anonymous namespace)::S3Client in libarrow.a(s3fs.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[179/185] Linking CXX shared library release/libgandiva_jni.900.0.0.dylib
ninja: build stopped: subcommand failed.
ec2-user@ip-x-x-x-x Desktop %

I can see those simbols in libarrow.a, for example, GetObjectAttributes:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/release/libarrow.a | grep GetObjectAttributes
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
no symbols
U __ZNK3Aws2S38S3Client19GetObjectAttributesERKNS0_5Model26GetObjectAttributesRequestE
U __ZNK3Aws2S38S3Client24GetObjectAttributesAsyncERKNS0_5Model26GetObjectAttributesRequestERKNSt3__18functionIFvPKS1_S5_RKNS_5Utils7OutcomeINS2_25GetObjectAttributesResultENS0_7S3ErrorEEERKNS6_10shared_ptrIKNS_6Client18AsyncCallerContextEEEEEESN_
U __ZNK3Aws2S38S3Client27GetObjectAttributesCallableERKNS0_5Model26GetObjectAttributesRequestE

It is in the source code: S3Client.h#L2563 and S3Client.cpp#L1917 but cannot be found in the symbols:
ec2-user@ip-x-x-x-x arrow % nm ./cpp-build/cpp/awssdk_ep-prefix/src/awssdk_ep-build/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.a | grep GetObjectAttributes
ec2-user@ip-x-x-x-x arrow %

@kou any thoughts?
brew's version for aws-sdk-cpp is 1.9.300
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 106c8166c2d29f68c022fe477babea0f9e82950f
Submitted crossbow builds: ursacomputing/crossbow @ actions-cfd8fcd94f
TaskStatus
java-jarsGithub Actions
@kou
kou
commented
Jul 28, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 28, 2022

Copy link


Copy Markdown

Revision: 1ac0510ec9f714c092c2b37c75542c81990adccd
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
TaskStatus
java-jarsGithub Actions
REASYand others
added 11 commits
July 28, 2022 20:03
@REASY
feat: Java JNI with S3 support
4ea0d4b
@REASY
feat: Java JNI with S3 support
dcab461
@REASY
feat: Arrow S3 is controlled by variable
e99fb28
@REASY
fix: Alphabetical order
f13b5f8
@REASY
fix: Link AWS SDK C++ statically, otherwise getting `Error: Unexpecte…
2afc96f
…d shared dependency found in libgandiva_jni.dylib: `libaws-cpp-sdk-config`
`
@REASY
fix: Use MacOS 10.13, otherwise getting `/Users/runner/work/crossbow/…
639d0b4
…crossbow/arrow/cpp-build/awssdk_ep-prefix/src/awssdk_ep/aws-cpp-sdk-core/source/utils/crypto/commoncrypto/CryptoImpl.cpp:505:27: error: 'CCCryptorGCMSetIV' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]`
@kou@REASY
Fix a typo
fbaf5f0
@kou@REASY
Allow libcurl
8f848a1
@kou@REASY
Allow CoreFoundation
b09a68c
@kou@REASY
Make verboase
da354a9
@kou@REASY
Uninstall Homebrew's aws-sdk-cpp to avoid mixing
3c7255a
@REASY
REASYforce-pushed
the
feature/java-with-s3
branch
from
f3c2263 to
3c7255aCompareJuly 28, 2022 12:04
@REASY
REASY
commented
Jul 28, 2022

Copy link


Copy Markdown

ContributorAuthor
Revision: 1ac0510
Submitted crossbow builds: ursacomputing/crossbow @ actions-683460ca62
Task	Status

java-jars	Github Actions
This should be fixed by #13712
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
@github-actions crossbow submit java-jars
@github-actions
github-actionsBot
commented
Jul 29, 2022

Copy link


Copy Markdown

Revision: 3c7255a
Submitted crossbow builds: ursacomputing/crossbow @ actions-ea49ddbb96
TaskStatus
java-jarsGithub Actions
kou
kou
approved these changes
Jul 29, 2022
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Thanks!
@kou
kou
merged commit 7fe71f5
into
apache:masterJul 29, 2022
@REASY
REASY
commented
Jul 29, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Thanks, @kou!
Could you, please, explain why it could not find the symbols? I saw you removed brew's version of aws-sdk-cpp. In ThirpartyToolChain.cmake I can see
 elseif(AWSSDK_SOURCE STREQUAL "BUNDLED")
build_awssdk()

Wasn't it enough for cmake to use built version?
@kou
kou
commented
Jul 29, 2022

Copy link


Copy Markdown

Member
Sure.
Our build used header files provided by aws-sdk-cpp installed by Homebrew but used library files provided by our bundled aws-sdk-cpp. It causes the problem.
Our build used packages installed by Homebrew. Our build adds -I$(brew --prefix)/include for them. aws-sdk-cpp installed by Homebrew installs it's header files to $(brew --prefix)/include. Our build used bundled aws-sdk-cpp but header files provided by aws-sdk-cpp installed by Homebrew are used because header file search path includes $(brew --prefix)/include before header file search path for bundled aws-sdk-cpp.
@ursabot
ursabot
commented
Jul 29, 2022

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = fa7e7a3 and contender = 7fe71f5. 7fe71f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.78% ⬆️0.11%] ursa-thinkcentre-m75q

Buildkite builds:

[Failed] 7fe71f5c ec2-t3-xlarge-us-east-2

[Finished] 7fe71f5c test-mac-arm

[Finished] 7fe71f5c ursa-i9-9960x

[Finished] 7fe71f5c ursa-thinkcentre-m75q

[Failed] fa7e7a3f ec2-t3-xlarge-us-east-2

[Failed] fa7e7a3f test-mac-arm

[Finished] fa7e7a3f ursa-i9-9960x

[Finished] fa7e7a3f ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@koukou
mentioned this pull request
Aug 19, 2022
 Merged

@asfimportasfimport
mentioned this pull request
Nov 26, 2024

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@REASY@kou@raulcd@pitrou@adzcai@ursabot