GH-43164: [C++] Fix CMake link order for AWS SDK - #43230

Merged
kou merged 1 commit into
apache:mainfrom
llama90:ARROW-43164
Jul 15, 2024
Merged

GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom
llama90:ARROW-43164

Conversation

@llama90

@llama90llama90 commented Jul 11, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.

What changes are included in this PR?

  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.

Are these changes tested?

I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.

Are there any user-facing changes?

No.

@github-actionsgithub-actionsBot added the awaiting review Awaiting review label Jul 11, 2024
@github-actions

github-actionsBot commented Jul 11, 2024

Copy link
Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.

@llama90llama90 changed the title GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90

llama90 commented Jul 13, 2024

Copy link
Copy Markdown
ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90

llama90 commented Jul 13, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@kou Hello. Could you review this when your free time? Thank you.

I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:

I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.

arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955 in 7184150

// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile"));

I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.

However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.

@kou

kou commented Jul 13, 2024

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g cpp java-jars

@kou

kou commented Jul 13, 2024

Copy link
Copy Markdown
Member

I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?

Could you open an issue (or issues?) for them?

I'm curious why the S3 tests are excluded in these environments and how to address this issue.

Is this issue correct? It seems that it's unrelated...

@github-actions

This comment was marked as outdated.

Sign in to view
@kou

kou commented Jul 13, 2024

Copy link
Copy Markdown
Member

Could you rebase on main for e59832f ?

@llama90

llama90 commented Jul 14, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Could you rebase on main for e59832f ?

Done!

Is this issue correct? It seems that it's unrelated...

Ah...! I reviewed this again.

Could you open an issue (or issues?) for them?

I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.

Thank you for your review!

@kou

kou commented Jul 14, 2024

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g cpp java-jars

@github-actions

github-actionsBot commented Jul 14, 2024

Copy link
Copy Markdown

Revision: f567995

Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8

TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions

kou
kou approved these changes Jul 15, 2024
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@kou
kou merged commit 57ac40c into apache:mainJul 15, 2024
@koukou removed the awaiting review Awaiting review label Jul 15, 2024
@koukou mentioned this pull request Jul 15, 2024
Closed
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Jul 15, 2024
@conbench-apache-arrow

conbench-apache-arrowBot commented Jul 15, 2024

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

@vibhatha

vibhatha commented Jul 24, 2024

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g java

@github-actions

github-actionsBot commented Jul 24, 2024

Copy link
Copy Markdown

Revision: f567995

Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9

TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees

No one assigned

Labels

awaiting mergeAwaiting mergeComponent: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-43164: [C++] Fix CMake link order for AWS SDK - #43230
Merged
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Jul 15, 2024
Merged
GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Conversation
@llama90
@llama90llama90
commented
Jul 11, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.
What changes are included in this PR?
  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.
Are these changes tested?
I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.
Are there any user-facing changes?
No.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jul 11, 2024
@github-actions
github-actionsBot
commented
Jul 11, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.
@llama90llama90
changed the title
GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90
llama90
commented
Jul 13, 2024

Copy link


Copy Markdown

ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90
llama90
commented
Jul 13, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou Hello. Could you review this when your free time? Thank you.
I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.
arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955
in
7184150
// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile")); 
I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.
However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?
Could you open an issue (or issues?) for them?
I'm curious why the S3 tests are excluded in these environments and how to address this issue.
Is this issue correct? It seems that it's unrelated...
@github-actions

This comment was marked as outdated.

Sign in to view
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
Could you rebase on main for e59832f ?
@llama90
llama90
commented
Jul 14, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Could you rebase on main for e59832f ?
Done!
Is this issue correct? It seems that it's unrelated...
Ah...! I reviewed this again.
Could you open an issue (or issues?) for them?
I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.
Thank you for your review!
@kou
kou
commented
Jul 14, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@github-actions
github-actionsBot
commented
Jul 14, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8
TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
kou
kou
approved these changes
Jul 15, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit 57ac40c
into
apache:mainJul 15, 2024
@koukou
removed
the 
awaiting review
Awaiting review
label
Jul 15, 2024
@koukou
mentioned this pull request
Jul 15, 2024

Closed

@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Jul 15, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 15, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@vibhatha
vibhatha
commented
Jul 24, 2024

Copy link


Copy Markdown

Contributor
@github-actions crossbow submit -g java
@github-actions
github-actionsBot
commented
Jul 24, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9
TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-43164: [C++] Fix CMake link order for AWS SDK - #43230
Merged
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Jul 15, 2024
Merged
GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Conversation
@llama90
@llama90llama90
commented
Jul 11, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.
What changes are included in this PR?
  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.
Are these changes tested?
I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.
Are there any user-facing changes?
No.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jul 11, 2024
@github-actions
github-actionsBot
commented
Jul 11, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.
@llama90llama90
changed the title
GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90
llama90
commented
Jul 13, 2024

Copy link


Copy Markdown

ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90
llama90
commented
Jul 13, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou Hello. Could you review this when your free time? Thank you.
I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.
arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955
in
7184150
// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile")); 
I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.
However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?
Could you open an issue (or issues?) for them?
I'm curious why the S3 tests are excluded in these environments and how to address this issue.
Is this issue correct? It seems that it's unrelated...
@github-actions

This comment was marked as outdated.

Sign in to view
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
Could you rebase on main for e59832f ?
@llama90
llama90
commented
Jul 14, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Could you rebase on main for e59832f ?
Done!
Is this issue correct? It seems that it's unrelated...
Ah...! I reviewed this again.
Could you open an issue (or issues?) for them?
I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.
Thank you for your review!
@kou
kou
commented
Jul 14, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@github-actions
github-actionsBot
commented
Jul 14, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8
TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
kou
kou
approved these changes
Jul 15, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit 57ac40c
into
apache:mainJul 15, 2024
@koukou
removed
the 
awaiting review
Awaiting review
label
Jul 15, 2024
@koukou
mentioned this pull request
Jul 15, 2024

Closed

@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Jul 15, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 15, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@vibhatha
vibhatha
commented
Jul 24, 2024

Copy link


Copy Markdown

Contributor
@github-actions crossbow submit -g java
@github-actions
github-actionsBot
commented
Jul 24, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9
TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-43164: [C++] Fix CMake link order for AWS SDK - #43230
Merged
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Jul 15, 2024
Merged
GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Conversation
@llama90
@llama90llama90
commented
Jul 11, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.
What changes are included in this PR?
  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.
Are these changes tested?
I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.
Are there any user-facing changes?
No.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jul 11, 2024
@github-actions
github-actionsBot
commented
Jul 11, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.
@llama90llama90
changed the title
GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90
llama90
commented
Jul 13, 2024

Copy link


Copy Markdown

ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90
llama90
commented
Jul 13, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou Hello. Could you review this when your free time? Thank you.
I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.
arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955
in
7184150
// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile")); 
I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.
However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?
Could you open an issue (or issues?) for them?
I'm curious why the S3 tests are excluded in these environments and how to address this issue.
Is this issue correct? It seems that it's unrelated...
@github-actions

This comment was marked as outdated.

Sign in to view
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
Could you rebase on main for e59832f ?
@llama90
llama90
commented
Jul 14, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Could you rebase on main for e59832f ?
Done!
Is this issue correct? It seems that it's unrelated...
Ah...! I reviewed this again.
Could you open an issue (or issues?) for them?
I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.
Thank you for your review!
@kou
kou
commented
Jul 14, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@github-actions
github-actionsBot
commented
Jul 14, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8
TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
kou
kou
approved these changes
Jul 15, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit 57ac40c
into
apache:mainJul 15, 2024
@koukou
removed
the 
awaiting review
Awaiting review
label
Jul 15, 2024
@koukou
mentioned this pull request
Jul 15, 2024

Closed

@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Jul 15, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 15, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@vibhatha
vibhatha
commented
Jul 24, 2024

Copy link


Copy Markdown

Contributor
@github-actions crossbow submit -g java
@github-actions
github-actionsBot
commented
Jul 24, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9
TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-43164: [C++] Fix CMake link order for AWS SDK - #43230
Merged
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Jul 15, 2024
Merged
GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Conversation
@llama90
@llama90llama90
commented
Jul 11, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.
What changes are included in this PR?
  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.
Are these changes tested?
I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.
Are there any user-facing changes?
No.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jul 11, 2024
@github-actions
github-actionsBot
commented
Jul 11, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.
@llama90llama90
changed the title
GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90
llama90
commented
Jul 13, 2024

Copy link


Copy Markdown

ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90
llama90
commented
Jul 13, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou Hello. Could you review this when your free time? Thank you.
I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.
arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955
in
7184150
// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile")); 
I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.
However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?
Could you open an issue (or issues?) for them?
I'm curious why the S3 tests are excluded in these environments and how to address this issue.
Is this issue correct? It seems that it's unrelated...
@github-actions

This comment was marked as outdated.

Sign in to view
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
Could you rebase on main for e59832f ?
@llama90
llama90
commented
Jul 14, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Could you rebase on main for e59832f ?
Done!
Is this issue correct? It seems that it's unrelated...
Ah...! I reviewed this again.
Could you open an issue (or issues?) for them?
I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.
Thank you for your review!
@kou
kou
commented
Jul 14, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@github-actions
github-actionsBot
commented
Jul 14, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8
TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
kou
kou
approved these changes
Jul 15, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit 57ac40c
into
apache:mainJul 15, 2024
@koukou
removed
the 
awaiting review
Awaiting review
label
Jul 15, 2024
@koukou
mentioned this pull request
Jul 15, 2024

Closed

@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Jul 15, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 15, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@vibhatha
vibhatha
commented
Jul 24, 2024

Copy link


Copy Markdown

Contributor
@github-actions crossbow submit -g java
@github-actions
github-actionsBot
commented
Jul 24, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9
TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-43164: [C++] Fix CMake link order for AWS SDK - #43230
Merged
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Jul 15, 2024
Merged
GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Conversation
@llama90
@llama90llama90
commented
Jul 11, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.
What changes are included in this PR?
  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.
Are these changes tested?
I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.
Are there any user-facing changes?
No.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jul 11, 2024
@github-actions
github-actionsBot
commented
Jul 11, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.
@llama90llama90
changed the title
GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90
llama90
commented
Jul 13, 2024

Copy link


Copy Markdown

ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90
llama90
commented
Jul 13, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou Hello. Could you review this when your free time? Thank you.
I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.
arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955
in
7184150
// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile")); 
I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.
However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?
Could you open an issue (or issues?) for them?
I'm curious why the S3 tests are excluded in these environments and how to address this issue.
Is this issue correct? It seems that it's unrelated...
@github-actions

This comment was marked as outdated.

Sign in to view
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
Could you rebase on main for e59832f ?
@llama90
llama90
commented
Jul 14, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Could you rebase on main for e59832f ?
Done!
Is this issue correct? It seems that it's unrelated...
Ah...! I reviewed this again.
Could you open an issue (or issues?) for them?
I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.
Thank you for your review!
@kou
kou
commented
Jul 14, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@github-actions
github-actionsBot
commented
Jul 14, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8
TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
kou
kou
approved these changes
Jul 15, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit 57ac40c
into
apache:mainJul 15, 2024
@koukou
removed
the 
awaiting review
Awaiting review
label
Jul 15, 2024
@koukou
mentioned this pull request
Jul 15, 2024

Closed

@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Jul 15, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 15, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@vibhatha
vibhatha
commented
Jul 24, 2024

Copy link


Copy Markdown

Contributor
@github-actions crossbow submit -g java
@github-actions
github-actionsBot
commented
Jul 24, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9
TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-43164: [C++] Fix CMake link order for AWS SDK - #43230
Merged
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Jul 15, 2024
Merged
GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Conversation
@llama90
@llama90llama90
commented
Jul 11, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.
What changes are included in this PR?
  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.
Are these changes tested?
I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.
Are there any user-facing changes?
No.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jul 11, 2024
@github-actions
github-actionsBot
commented
Jul 11, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.
@llama90llama90
changed the title
GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90
llama90
commented
Jul 13, 2024

Copy link


Copy Markdown

ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90
llama90
commented
Jul 13, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou Hello. Could you review this when your free time? Thank you.
I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.
arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955
in
7184150
// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile")); 
I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.
However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?
Could you open an issue (or issues?) for them?
I'm curious why the S3 tests are excluded in these environments and how to address this issue.
Is this issue correct? It seems that it's unrelated...
@github-actions

This comment was marked as outdated.

Sign in to view
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
Could you rebase on main for e59832f ?
@llama90
llama90
commented
Jul 14, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Could you rebase on main for e59832f ?
Done!
Is this issue correct? It seems that it's unrelated...
Ah...! I reviewed this again.
Could you open an issue (or issues?) for them?
I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.
Thank you for your review!
@kou
kou
commented
Jul 14, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@github-actions
github-actionsBot
commented
Jul 14, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8
TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
kou
kou
approved these changes
Jul 15, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit 57ac40c
into
apache:mainJul 15, 2024
@koukou
removed
the 
awaiting review
Awaiting review
label
Jul 15, 2024
@koukou
mentioned this pull request
Jul 15, 2024

Closed

@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Jul 15, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 15, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@vibhatha
vibhatha
commented
Jul 24, 2024

Copy link


Copy Markdown

Contributor
@github-actions crossbow submit -g java
@github-actions
github-actionsBot
commented
Jul 24, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9
TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-43164: [C++] Fix CMake link order for AWS SDK - #43230
Merged
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Jul 15, 2024
Merged
GH-43164: [C++] Fix CMake link order for AWS SDK#43230
kou merged 1 commit into
apache:mainfrom 
llama90:ARROW-43164
Conversation
@llama90
@llama90llama90
commented
Jul 11, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
To resolve conflicts with AWS SDK for C++ static variables when linked with libarrow by ensuring correct link order.
What changes are included in this PR?
  • Adjusted CMakeLists.txt to set ARROW_S3_TEST_EXTRA_LINK_LIBS.
  • Ensured libarrow is linked before libaws* libraries.
  • Updated s3fs_test configuration to use the new link order.
Are these changes tested?
I ran the test locally and observed the same result as mentioned. Additionally, I confirmed that if ARROW_S3 is set to OFF or if the configuration includes exclude_tests=arrow-s3fs-test, the test is excluded.
Are there any user-facing changes?
No.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Jul 11, 2024
@github-actions
github-actionsBot
commented
Jul 11, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #43164has been automatically assigned in GitHub to PR creator.
@llama90llama90
changed the title
GH-43164: [C++] Fix CMake link order to prevent AWS SDK for C++ static variable conflictGH-43164: [C++] Fix CMake link order for AWS SDKJul 13, 2024
@llama90
llama90
commented
Jul 13, 2024

Copy link


Copy Markdown

ContributorAuthor
arrow-s3fs-test result
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (903 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (906 ms total)
70: 70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (1327 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (37 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (31 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1400 ms total)
70: 70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:395: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70: 70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (780 ms)
70: [----------] 1 test from TestMinioServer (780 ms total)
70: 70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (30 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (33 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (80 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (77 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (68 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (78 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (8646 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (28 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:955: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (71 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (37 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (70 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (72 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (58 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (85 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (73 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (42 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (46 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (68 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (631 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (565 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (38 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (33 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (36 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (47 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (70 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (71 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (77 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (72 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (110 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (33 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (59 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (59 ms)
70: [----------] 32 tests from TestS3FS (11580 ms total)
70: 70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (37 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (25 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (117 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (150 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (133 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (48 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (70 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (104 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (164 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (18 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (94 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (64 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (104 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (107 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (67 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (72 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (95 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (23 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (85 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (80 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (79 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (71 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (44 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (58 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (79 ms)
70: [----------] 26 tests from TestS3FSGeneric (2083 ms total)
70: 70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70: 70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (16795 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70: 70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Failed

@llama90
llama90
commented
Jul 13, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou Hello. Could you review this when your free time? Thank you.
I am using an M1 MacBook, and I noticed that my local tests are failing while the CI tests are passing. I found that certain environments are not running the tests for ARROW_S3, including the following:
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation? I'm curious why the S3 tests are excluded in these environments and how to address this issue.
arrow/cpp/src/arrow/filesystem/s3fs_test.cc

Lines 954 to 955
in
7184150
// Existing "file", should fail
ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile")); 
I suspect that some tests might be designed to test the file system semantics on object storage (MinIO). For example, in object storage, uploading an object (file) with the same prefix (folder) often does not check for pre-existence and may manage via versioning or overwrite the existing object.
However, upon examining the tests, they seem to fail when a file already exists, but because MinIO is used internally for testing, it operates with object storage semantics, leading to test failures.
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
I encountered some failing test cases. Is it safe to ignore these for now? How should I proceed to improve the situation?
Could you open an issue (or issues?) for them?
I'm curious why the S3 tests are excluded in these environments and how to address this issue.
Is this issue correct? It seems that it's unrelated...
@github-actions

This comment was marked as outdated.

Sign in to view
@kou
kou
commented
Jul 13, 2024

Copy link


Copy Markdown

Member
Could you rebase on main for e59832f ?
@llama90
llama90
commented
Jul 14, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Could you rebase on main for e59832f ?
Done!
Is this issue correct? It seems that it's unrelated...
Ah...! I reviewed this again.
Could you open an issue (or issues?) for them?
I have identified various issues such as the Minio version and timeout problems. It seems the issue is related to various factors. Let's examine each one in detail. If necessary, I will create a new issue.
Thank you for your review!
@kou
kou
commented
Jul 14, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g cpp java-jars
@github-actions
github-actionsBot
commented
Jul 14, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-c9ef5bb9e8
TaskStatus
java-jarsGitHub Actions
test-alpine-linux-cppGitHub Actions
test-build-cpp-fuzzGitHub Actions
test-conda-cppGitHub Actions
test-conda-cpp-valgrindGitHub Actions
test-cuda-cppGitHub Actions
test-debian-12-cpp-amd64GitHub Actions
test-debian-12-cpp-i386GitHub Actions
test-fedora-39-cppGitHub Actions
test-ubuntu-20.04-cppGitHub Actions
test-ubuntu-20.04-cpp-bundledGitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formatsGitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGitHub Actions
test-ubuntu-22.04-cppGitHub Actions
test-ubuntu-22.04-cpp-20GitHub Actions
test-ubuntu-22.04-cpp-emscriptenGitHub Actions
test-ubuntu-22.04-cpp-no-threadingGitHub Actions
test-ubuntu-24.04-cppGitHub Actions
test-ubuntu-24.04-cpp-gcc-14GitHub Actions
kou
kou
approved these changes
Jul 15, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit 57ac40c
into
apache:mainJul 15, 2024
@koukou
removed
the 
awaiting review
Awaiting review
label
Jul 15, 2024
@koukou
mentioned this pull request
Jul 15, 2024

Closed

@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Jul 15, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Jul 15, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57ac40c.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
@vibhatha
vibhatha
commented
Jul 24, 2024

Copy link


Copy Markdown

Contributor
@github-actions crossbow submit -g java
@github-actions
github-actionsBot
commented
Jul 24, 2024

Copy link


Copy Markdown

Revision: f567995
Submitted crossbow builds: ursacomputing/crossbow @ actions-e631314dd9
TaskStatus
java-jarsGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@llama90@kou@vibhatha