GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080

Merged
kou merged 11 commits into
apache:mainfrom
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged

GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Feb 14, 2024•
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

We want python side tests of AzureFileSystem to run in CI.

What changes are included in this PR?

  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121

Probably all of this should have been included in #39971

Are these changes tested?

There is no new functionality to test

Are there any user-facing changes?

No

@github-actions

github-actionsBot commented Feb 14, 2024

Copy link
Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.

@github-actionsgithub-actionsBot added the awaiting review Awaiting review label Feb 14, 2024
@Tom-NewtonTom-Newton mentioned this pull request Feb 14, 2024
Merged
@Tom-Newton

Tom-Newton commented Feb 14, 2024

Copy link
Copy Markdown
ContributorAuthor

Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.

I will need to investigate the AVX2 build failure.

kou
kou reviewed Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile Outdated
Comment on lines +25 to +30
# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
apt-get update -y -q && \
apt-get install -y -q npm \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

@koukouFeb 14, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?

@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Feb 14, 2024
@kou

kou commented Feb 14, 2024

Copy link
Copy Markdown
Member

Test is AzureFileSsstem.InitializeWithDefaultCredential:

https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540

[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291 in 2422994

TEST(AzureFileSystem, InitializeWithDefaultCredential) {
AzureOptions options;
options.account_name = "dummy-account-name";
ARROW_EXPECT_OK(options.ConfigureDefaultCredential());
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options));
}

Important backtrace:

https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568

Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185 in 2422994

return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name),
token_credential_);

@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 14, 2024
@Tom-Newton

Tom-Newton commented Feb 17, 2024

Copy link
Copy Markdown
ContributorAuthor

Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.

@kou

kou commented Feb 17, 2024

Copy link
Copy Markdown
Member

Pinning to same versions we get from vcpkg solves some of the CI failures.

Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)
Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?

Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?

I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.

Could you open an issue for it so that we can work on it as a separated task?

@Tom-Newton

Tom-Newton commented Feb 18, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Created an issue for the python 3.9 GCS testbench issue #40112

For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322

and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels

If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.

@Tom-Newton
Tom-Newton marked this pull request as ready for review February 18, 2024 12:25
@Tom-Newton

Tom-Newton commented Feb 18, 2024

Copy link
Copy Markdown
ContributorAuthor

I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.

@Tom-Newton

Tom-Newton commented Feb 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.
Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614
Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080

C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.

I've run this multiple times locally but so far have not been able to reproduce.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/more_azure_builds_and_tests/GH-40079 branch from 69ed3a5 to 7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton

Tom-Newton commented Feb 18, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080

@Tom-Newton

Tom-Newton commented Feb 18, 2024

Copy link
Copy Markdown
ContributorAuthor

I think #40120 should resolve the Python 3.9 and GCS testbench problem.

I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.

I created a separate issue to fix this flakiness #40121

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/more_azure_builds_and_tests/GH-40079 branch from 7ed0910 to 7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/more_azure_builds_and_tests/GH-40079 branch from 31e4920 to 5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche reviewed Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt Outdated
# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911

@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).

But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?

@Tom-NewtonTom-NewtonFeb 20, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.

@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?

@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.

It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.

@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Feb 20, 2024
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 21, 2024
@assignUser

assignUser commented Feb 22, 2024

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g python

@github-actions

github-actionsBot commented Feb 22, 2024

Copy link
Copy Markdown

Revision: 53b9d2c

Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8

TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions

@Tom-Newton

Tom-Newton commented Feb 22, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Remaining CI failures:
s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930
appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages
A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409

I don't think any of these are related to my changes.

assignUser
assignUser approved these changes Feb 22, 2024
View reviewed changes

@assignUserassignUser left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)

Comment threadci/scripts/cpp_test.sh
@kou

kou commented Feb 22, 2024

Copy link
Copy Markdown
Member

Those failures are unrelated because they are happen on main too.

kou
kou approved these changes Feb 22, 2024
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@kou
kou merged commit b089c6a into apache:mainFeb 22, 2024
@koukou removed the awaiting change review Awaiting change review label Feb 22, 2024
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Feb 22, 2024
@conbench-apache-arrow

conbench-apache-arrowBot commented Feb 23, 2024

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

@booxterbooxter mentioned this pull request Aug 15, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees

No one assigned

Labels

awaiting mergeAwaiting merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds

4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080
Merged
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged
GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Feb 14, 2024•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We want python side tests of AzureFileSystem to run in CI.
What changes are included in this PR?
  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121
Probably all of this should have been included in #39971
Are these changes tested?
There is no new functionality to test
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Feb 14, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Feb 14, 2024
@Tom-NewtonTom-Newton
mentioned this pull request
Feb 14, 2024
 Merged

@Tom-Newton
Tom-Newton
commented
Feb 14, 2024

Copy link


Copy Markdown

ContributorAuthor
Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.
I will need to investigate the AVX2 build failure.
kou
kou
reviewed
Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile
Outdated


Comment on lines

+25
 to 
+30

# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
 apt-get update -y -q && \
 apt-get install -y -q npm \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*
@koukouFeb 14, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Feb 14, 2024
@kou
kou
commented
Feb 14, 2024

Copy link


Copy Markdown

Member
Test is AzureFileSsstem.InitializeWithDefaultCredential:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540
[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291
in
2422994
TEST(AzureFileSystem, InitializeWithDefaultCredential) { 
 AzureOptions options; 
 options.account_name = "dummy-account-name"; 
ARROW_EXPECT_OK(options.ConfigureDefaultCredential()); 
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options)); 
 } 
Important backtrace:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568
Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185
in
2422994
return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name), 
 token_credential_); 
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 14, 2024
@Tom-Newton
Tom-Newton
commented
Feb 17, 2024

Copy link


Copy Markdown

ContributorAuthor
Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
@kou
kou
commented
Feb 17, 2024

Copy link


Copy Markdown

Member
Pinning to same versions we get from vcpkg solves some of the CI failures.
Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)

Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?
Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?
I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
Could you open an issue for it so that we can work on it as a separated task?
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Created an issue for the python 3.9 GCS testbench issue #40112
For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322
and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels
If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
February 18, 2024 12:25
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.

Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614

Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.
I've run this multiple times locally but so far have not been able to reproduce.
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
69ed3a5 to
7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I think #40120 should resolve the Python 3.9 and GCS testbench problem.
I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.
I created a separate issue to fix this flakiness #40121
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
7ed0910 to
7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
31e4920 to
5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche
reviewed
Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt
Outdated

# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).
But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?
@Tom-NewtonTom-NewtonFeb 20, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.
@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?
@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.
It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Feb 20, 2024
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 21, 2024
@assignUser
assignUser
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python
@github-actions
github-actionsBot
commented
Feb 22, 2024

Copy link


Copy Markdown

Revision: 53b9d2c
Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8
TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions
@Tom-Newton
Tom-Newton
commented
Feb 22, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Remaining CI failures:

s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930

appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages

A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409
I don't think any of these are related to my changes.
assignUser
assignUser
approved these changes
Feb 22, 2024
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)
Comment threadci/scripts/cpp_test.sh
@kou
kou
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
Those failures are unrelated because they are happen on main too.
kou
kou
approved these changes
Feb 22, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit b089c6a
into
apache:mainFeb 22, 2024
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Feb 22, 2024
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Feb 22, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 23, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
@booxterbooxter
mentioned this pull request
Aug 15, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds


4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080
Merged
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged
GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Feb 14, 2024•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We want python side tests of AzureFileSystem to run in CI.
What changes are included in this PR?
  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121
Probably all of this should have been included in #39971
Are these changes tested?
There is no new functionality to test
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Feb 14, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Feb 14, 2024
@Tom-NewtonTom-Newton
mentioned this pull request
Feb 14, 2024
 Merged

@Tom-Newton
Tom-Newton
commented
Feb 14, 2024

Copy link


Copy Markdown

ContributorAuthor
Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.
I will need to investigate the AVX2 build failure.
kou
kou
reviewed
Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile
Outdated


Comment on lines

+25
 to 
+30

# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
 apt-get update -y -q && \
 apt-get install -y -q npm \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*
@koukouFeb 14, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Feb 14, 2024
@kou
kou
commented
Feb 14, 2024

Copy link


Copy Markdown

Member
Test is AzureFileSsstem.InitializeWithDefaultCredential:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540
[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291
in
2422994
TEST(AzureFileSystem, InitializeWithDefaultCredential) { 
 AzureOptions options; 
 options.account_name = "dummy-account-name"; 
ARROW_EXPECT_OK(options.ConfigureDefaultCredential()); 
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options)); 
 } 
Important backtrace:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568
Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185
in
2422994
return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name), 
 token_credential_); 
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 14, 2024
@Tom-Newton
Tom-Newton
commented
Feb 17, 2024

Copy link


Copy Markdown

ContributorAuthor
Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
@kou
kou
commented
Feb 17, 2024

Copy link


Copy Markdown

Member
Pinning to same versions we get from vcpkg solves some of the CI failures.
Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)

Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?
Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?
I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
Could you open an issue for it so that we can work on it as a separated task?
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Created an issue for the python 3.9 GCS testbench issue #40112
For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322
and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels
If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
February 18, 2024 12:25
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.

Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614

Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.
I've run this multiple times locally but so far have not been able to reproduce.
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
69ed3a5 to
7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I think #40120 should resolve the Python 3.9 and GCS testbench problem.
I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.
I created a separate issue to fix this flakiness #40121
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
7ed0910 to
7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
31e4920 to
5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche
reviewed
Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt
Outdated

# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).
But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?
@Tom-NewtonTom-NewtonFeb 20, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.
@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?
@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.
It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Feb 20, 2024
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 21, 2024
@assignUser
assignUser
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python
@github-actions
github-actionsBot
commented
Feb 22, 2024

Copy link


Copy Markdown

Revision: 53b9d2c
Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8
TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions
@Tom-Newton
Tom-Newton
commented
Feb 22, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Remaining CI failures:

s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930

appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages

A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409
I don't think any of these are related to my changes.
assignUser
assignUser
approved these changes
Feb 22, 2024
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)
Comment threadci/scripts/cpp_test.sh
@kou
kou
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
Those failures are unrelated because they are happen on main too.
kou
kou
approved these changes
Feb 22, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit b089c6a
into
apache:mainFeb 22, 2024
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Feb 22, 2024
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Feb 22, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 23, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
@booxterbooxter
mentioned this pull request
Aug 15, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds


4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080
Merged
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged
GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Feb 14, 2024•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We want python side tests of AzureFileSystem to run in CI.
What changes are included in this PR?
  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121
Probably all of this should have been included in #39971
Are these changes tested?
There is no new functionality to test
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Feb 14, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Feb 14, 2024
@Tom-NewtonTom-Newton
mentioned this pull request
Feb 14, 2024
 Merged

@Tom-Newton
Tom-Newton
commented
Feb 14, 2024

Copy link


Copy Markdown

ContributorAuthor
Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.
I will need to investigate the AVX2 build failure.
kou
kou
reviewed
Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile
Outdated


Comment on lines

+25
 to 
+30

# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
 apt-get update -y -q && \
 apt-get install -y -q npm \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*
@koukouFeb 14, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Feb 14, 2024
@kou
kou
commented
Feb 14, 2024

Copy link


Copy Markdown

Member
Test is AzureFileSsstem.InitializeWithDefaultCredential:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540
[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291
in
2422994
TEST(AzureFileSystem, InitializeWithDefaultCredential) { 
 AzureOptions options; 
 options.account_name = "dummy-account-name"; 
ARROW_EXPECT_OK(options.ConfigureDefaultCredential()); 
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options)); 
 } 
Important backtrace:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568
Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185
in
2422994
return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name), 
 token_credential_); 
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 14, 2024
@Tom-Newton
Tom-Newton
commented
Feb 17, 2024

Copy link


Copy Markdown

ContributorAuthor
Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
@kou
kou
commented
Feb 17, 2024

Copy link


Copy Markdown

Member
Pinning to same versions we get from vcpkg solves some of the CI failures.
Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)

Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?
Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?
I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
Could you open an issue for it so that we can work on it as a separated task?
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Created an issue for the python 3.9 GCS testbench issue #40112
For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322
and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels
If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
February 18, 2024 12:25
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.

Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614

Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.
I've run this multiple times locally but so far have not been able to reproduce.
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
69ed3a5 to
7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I think #40120 should resolve the Python 3.9 and GCS testbench problem.
I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.
I created a separate issue to fix this flakiness #40121
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
7ed0910 to
7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
31e4920 to
5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche
reviewed
Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt
Outdated

# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).
But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?
@Tom-NewtonTom-NewtonFeb 20, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.
@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?
@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.
It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Feb 20, 2024
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 21, 2024
@assignUser
assignUser
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python
@github-actions
github-actionsBot
commented
Feb 22, 2024

Copy link


Copy Markdown

Revision: 53b9d2c
Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8
TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions
@Tom-Newton
Tom-Newton
commented
Feb 22, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Remaining CI failures:

s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930

appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages

A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409
I don't think any of these are related to my changes.
assignUser
assignUser
approved these changes
Feb 22, 2024
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)
Comment threadci/scripts/cpp_test.sh
@kou
kou
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
Those failures are unrelated because they are happen on main too.
kou
kou
approved these changes
Feb 22, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit b089c6a
into
apache:mainFeb 22, 2024
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Feb 22, 2024
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Feb 22, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 23, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
@booxterbooxter
mentioned this pull request
Aug 15, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds


4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080
Merged
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged
GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Feb 14, 2024•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We want python side tests of AzureFileSystem to run in CI.
What changes are included in this PR?
  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121
Probably all of this should have been included in #39971
Are these changes tested?
There is no new functionality to test
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Feb 14, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Feb 14, 2024
@Tom-NewtonTom-Newton
mentioned this pull request
Feb 14, 2024
 Merged

@Tom-Newton
Tom-Newton
commented
Feb 14, 2024

Copy link


Copy Markdown

ContributorAuthor
Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.
I will need to investigate the AVX2 build failure.
kou
kou
reviewed
Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile
Outdated


Comment on lines

+25
 to 
+30

# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
 apt-get update -y -q && \
 apt-get install -y -q npm \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*
@koukouFeb 14, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Feb 14, 2024
@kou
kou
commented
Feb 14, 2024

Copy link


Copy Markdown

Member
Test is AzureFileSsstem.InitializeWithDefaultCredential:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540
[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291
in
2422994
TEST(AzureFileSystem, InitializeWithDefaultCredential) { 
 AzureOptions options; 
 options.account_name = "dummy-account-name"; 
ARROW_EXPECT_OK(options.ConfigureDefaultCredential()); 
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options)); 
 } 
Important backtrace:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568
Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185
in
2422994
return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name), 
 token_credential_); 
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 14, 2024
@Tom-Newton
Tom-Newton
commented
Feb 17, 2024

Copy link


Copy Markdown

ContributorAuthor
Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
@kou
kou
commented
Feb 17, 2024

Copy link


Copy Markdown

Member
Pinning to same versions we get from vcpkg solves some of the CI failures.
Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)

Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?
Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?
I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
Could you open an issue for it so that we can work on it as a separated task?
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Created an issue for the python 3.9 GCS testbench issue #40112
For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322
and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels
If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
February 18, 2024 12:25
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.

Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614

Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.
I've run this multiple times locally but so far have not been able to reproduce.
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
69ed3a5 to
7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I think #40120 should resolve the Python 3.9 and GCS testbench problem.
I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.
I created a separate issue to fix this flakiness #40121
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
7ed0910 to
7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
31e4920 to
5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche
reviewed
Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt
Outdated

# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).
But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?
@Tom-NewtonTom-NewtonFeb 20, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.
@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?
@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.
It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Feb 20, 2024
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 21, 2024
@assignUser
assignUser
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python
@github-actions
github-actionsBot
commented
Feb 22, 2024

Copy link


Copy Markdown

Revision: 53b9d2c
Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8
TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions
@Tom-Newton
Tom-Newton
commented
Feb 22, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Remaining CI failures:

s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930

appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages

A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409
I don't think any of these are related to my changes.
assignUser
assignUser
approved these changes
Feb 22, 2024
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)
Comment threadci/scripts/cpp_test.sh
@kou
kou
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
Those failures are unrelated because they are happen on main too.
kou
kou
approved these changes
Feb 22, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit b089c6a
into
apache:mainFeb 22, 2024
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Feb 22, 2024
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Feb 22, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 23, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
@booxterbooxter
mentioned this pull request
Aug 15, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds


4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080
Merged
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged
GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Feb 14, 2024•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We want python side tests of AzureFileSystem to run in CI.
What changes are included in this PR?
  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121
Probably all of this should have been included in #39971
Are these changes tested?
There is no new functionality to test
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Feb 14, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Feb 14, 2024
@Tom-NewtonTom-Newton
mentioned this pull request
Feb 14, 2024
 Merged

@Tom-Newton
Tom-Newton
commented
Feb 14, 2024

Copy link


Copy Markdown

ContributorAuthor
Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.
I will need to investigate the AVX2 build failure.
kou
kou
reviewed
Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile
Outdated


Comment on lines

+25
 to 
+30

# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
 apt-get update -y -q && \
 apt-get install -y -q npm \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*
@koukouFeb 14, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Feb 14, 2024
@kou
kou
commented
Feb 14, 2024

Copy link


Copy Markdown

Member
Test is AzureFileSsstem.InitializeWithDefaultCredential:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540
[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291
in
2422994
TEST(AzureFileSystem, InitializeWithDefaultCredential) { 
 AzureOptions options; 
 options.account_name = "dummy-account-name"; 
ARROW_EXPECT_OK(options.ConfigureDefaultCredential()); 
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options)); 
 } 
Important backtrace:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568
Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185
in
2422994
return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name), 
 token_credential_); 
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 14, 2024
@Tom-Newton
Tom-Newton
commented
Feb 17, 2024

Copy link


Copy Markdown

ContributorAuthor
Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
@kou
kou
commented
Feb 17, 2024

Copy link


Copy Markdown

Member
Pinning to same versions we get from vcpkg solves some of the CI failures.
Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)

Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?
Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?
I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
Could you open an issue for it so that we can work on it as a separated task?
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Created an issue for the python 3.9 GCS testbench issue #40112
For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322
and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels
If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
February 18, 2024 12:25
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.

Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614

Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.
I've run this multiple times locally but so far have not been able to reproduce.
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
69ed3a5 to
7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I think #40120 should resolve the Python 3.9 and GCS testbench problem.
I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.
I created a separate issue to fix this flakiness #40121
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
7ed0910 to
7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
31e4920 to
5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche
reviewed
Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt
Outdated

# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).
But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?
@Tom-NewtonTom-NewtonFeb 20, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.
@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?
@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.
It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Feb 20, 2024
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 21, 2024
@assignUser
assignUser
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python
@github-actions
github-actionsBot
commented
Feb 22, 2024

Copy link


Copy Markdown

Revision: 53b9d2c
Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8
TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions
@Tom-Newton
Tom-Newton
commented
Feb 22, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Remaining CI failures:

s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930

appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages

A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409
I don't think any of these are related to my changes.
assignUser
assignUser
approved these changes
Feb 22, 2024
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)
Comment threadci/scripts/cpp_test.sh
@kou
kou
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
Those failures are unrelated because they are happen on main too.
kou
kou
approved these changes
Feb 22, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit b089c6a
into
apache:mainFeb 22, 2024
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Feb 22, 2024
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Feb 22, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 23, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
@booxterbooxter
mentioned this pull request
Aug 15, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds


4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080
Merged
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged
GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Feb 14, 2024•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We want python side tests of AzureFileSystem to run in CI.
What changes are included in this PR?
  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121
Probably all of this should have been included in #39971
Are these changes tested?
There is no new functionality to test
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Feb 14, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Feb 14, 2024
@Tom-NewtonTom-Newton
mentioned this pull request
Feb 14, 2024
 Merged

@Tom-Newton
Tom-Newton
commented
Feb 14, 2024

Copy link


Copy Markdown

ContributorAuthor
Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.
I will need to investigate the AVX2 build failure.
kou
kou
reviewed
Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile
Outdated


Comment on lines

+25
 to 
+30

# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
 apt-get update -y -q && \
 apt-get install -y -q npm \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*
@koukouFeb 14, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Feb 14, 2024
@kou
kou
commented
Feb 14, 2024

Copy link


Copy Markdown

Member
Test is AzureFileSsstem.InitializeWithDefaultCredential:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540
[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291
in
2422994
TEST(AzureFileSystem, InitializeWithDefaultCredential) { 
 AzureOptions options; 
 options.account_name = "dummy-account-name"; 
ARROW_EXPECT_OK(options.ConfigureDefaultCredential()); 
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options)); 
 } 
Important backtrace:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568
Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185
in
2422994
return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name), 
 token_credential_); 
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 14, 2024
@Tom-Newton
Tom-Newton
commented
Feb 17, 2024

Copy link


Copy Markdown

ContributorAuthor
Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
@kou
kou
commented
Feb 17, 2024

Copy link


Copy Markdown

Member
Pinning to same versions we get from vcpkg solves some of the CI failures.
Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)

Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?
Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?
I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
Could you open an issue for it so that we can work on it as a separated task?
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Created an issue for the python 3.9 GCS testbench issue #40112
For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322
and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels
If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
February 18, 2024 12:25
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.

Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614

Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.
I've run this multiple times locally but so far have not been able to reproduce.
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
69ed3a5 to
7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I think #40120 should resolve the Python 3.9 and GCS testbench problem.
I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.
I created a separate issue to fix this flakiness #40121
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
7ed0910 to
7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
31e4920 to
5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche
reviewed
Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt
Outdated

# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).
But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?
@Tom-NewtonTom-NewtonFeb 20, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.
@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?
@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.
It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Feb 20, 2024
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 21, 2024
@assignUser
assignUser
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python
@github-actions
github-actionsBot
commented
Feb 22, 2024

Copy link


Copy Markdown

Revision: 53b9d2c
Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8
TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions
@Tom-Newton
Tom-Newton
commented
Feb 22, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Remaining CI failures:

s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930

appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages

A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409
I don't think any of these are related to my changes.
assignUser
assignUser
approved these changes
Feb 22, 2024
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)
Comment threadci/scripts/cpp_test.sh
@kou
kou
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
Those failures are unrelated because they are happen on main too.
kou
kou
approved these changes
Feb 22, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit b089c6a
into
apache:mainFeb 22, 2024
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Feb 22, 2024
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Feb 22, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 23, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
@booxterbooxter
mentioned this pull request
Aug 15, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds


4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-40079: [CI][Packaging] Enable Azure in more tests and builds - #40080
Merged
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Feb 22, 2024
Merged
GH-40079: [CI][Packaging] Enable Azure in more tests and builds#40080
kou merged 11 commits into
apache:mainfrom 
Tom-Newton:tomnewton/more_azure_builds_and_tests/GH-40079
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Feb 14, 2024•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We want python side tests of AzureFileSystem to run in CI.
What changes are included in this PR?
  • Add missing export to enable Azure pyarrow tests
  • Enable azure in sdist tests.
  • Enable Azure on macos python builds
  • Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
  • Enable retries on C++ tests to mitigate [CI][FS][Azure] Azurite tests are flaking on main #40121
Probably all of this should have been included in #39971
Are these changes tested?
There is no new functionality to test
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Feb 14, 2024

Copy link


Copy Markdown

⚠️ GitHub issue #40079has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Feb 14, 2024
@Tom-NewtonTom-Newton
mentioned this pull request
Feb 14, 2024
 Merged

@Tom-Newton
Tom-Newton
commented
Feb 14, 2024

Copy link


Copy Markdown

ContributorAuthor
Most of the CI failures are actually reproducible on main with PYTHON=3.9 docker-compose build conda-python.
I will need to investigate the AVX2 build failure.
kou
kou
reviewed
Feb 14, 2024
View reviewed changes
Comment threadci/docker/conda-cpp.dockerfile
Outdated


Comment on lines

+25
 to 
+30

# Azurite requires npm
RUN export DEBIAN_FRONTEND=noninteractive && \
 apt-get update -y -q && \
 apt-get install -y -q npm \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*
@koukouFeb 14, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we install npm (Node.js) by conda?
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Feb 14, 2024
@kou
kou
commented
Feb 14, 2024

Copy link


Copy Markdown

Member
Test is AzureFileSsstem.InitializeWithDefaultCredential:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6540
[ RUN ] AzureFileSystem.InitializeWithDefaultCredential

arrow/cpp/src/arrow/filesystem/azurefs_test.cc

Lines 286 to 291
in
2422994
TEST(AzureFileSystem, InitializeWithDefaultCredential) { 
 AzureOptions options; 
 options.account_name = "dummy-account-name"; 
ARROW_EXPECT_OK(options.ConfigureDefaultCredential()); 
EXPECT_OK_AND_ASSIGN(auto fs, AzureFileSystem::Make(options)); 
 } 
Important backtrace:
https://github.com/apache/arrow/actions/runs/7902101938/job/21566998998?pr=40080#step:6:6568
Thread 1 (Thread 0x7f5c8e6d2cc0 (LWP 26205)):
#0 0x00007f5c92ddae4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f5c45596010, __str=...) at /home/conda/feedstock_root/build_artifacts/gcc_compilers_1706816862910/work/build/x86_64-conda-linux-gnu/libstdc++-v3/include/bits/basic_string.h:541
#1 0x000055e62c73586c in std::_Construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_construct.h:119
#2 0x000055e62c7292c5 in std::__do_uninit_copy<__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_uninitialized.h:120
#3 0x00007f5c91da6f70 in Azure::Storage::_internal::StorageBearerTokenAuthenticationPolicy::Clone() const () from /opt/conda/envs/arrow/lib/./libazure-storage-common.so
#4 0x00007f5c9307cfc9 in Azure::Storage::Blobs::BlobServiceClient::BlobServiceClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Azure::Core::Credentials::TokenCredential>, Azure::Storage::Blobs::BlobClientOptions const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so
#5 0x00007f5c95141a35 in std::make_unique<Azure::Storage::Blobs::BlobServiceClient, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Azure::Core::Credentials::TokenCredential>&> () at /opt/conda/envs/arrow/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_ptr.h:1065
#6 0x00007f5c95126c04 in arrow::fs::AzureOptions::MakeBlobServiceClient () at /arrow/cpp/src/arrow/filesystem/azurefs.cc:184

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 184 to 185
in
2422994
return std::make_unique<Blobs::BlobServiceClient>(AccountBlobUrl(account_name), 
 token_credential_); 
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 14, 2024
@Tom-Newton
Tom-Newton
commented
Feb 17, 2024

Copy link


Copy Markdown

ContributorAuthor
Pinning to same versions we get from vcpkg solves some of the CI failures. I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
@kou
kou
commented
Feb 17, 2024

Copy link


Copy Markdown

Member
Pinning to same versions we get from vcpkg solves some of the CI failures.
Does it mean that we may need to improve our Azure filesystem implementation to use more recent azure-sdk-for-cpp? (If it's needed, we should work on it as a separated task.)

Or does it mean that we have symbol conflict with azure-sdk-for-cpp installed by vcpkg and conda?
Anyway, could you add a comment why we need to pin azure-sdk-for-cpp?
I think the only remaining failures are ones that also exist on main. Specifically installing GCS testbench on python 3.9 seems to fail.
Could you open an issue for it so that we can work on it as a separated task?
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Created an issue for the python 3.9 GCS testbench issue #40112
For the segfault it looks like going from azure-core-cpp 1.10.3 to 1.11.0 causes the problem. I cannot reproduce when using the same version built from source. I also found a couple of github issue which looks very relevant
conda-forge/azure-core-cpp-feedstock#10 (comment)
Azure/azure-sdk-for-cpp#5322
and it looks like 1.11.0 has been marked as broken on the conda website
image
https://anaconda.org/conda-forge/azure-core-cpp/labels
If I understand this correctly it seems to be a conda issue not an Azure SDK or an Arrow problem.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
February 18, 2024 12:25
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I'm going to call this ready for review but I'm not sure if we can merge it while #40112 is un-resolved.
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
Hmm.... it looks like the azure tests are slightly flaky. I just changed a comment and the new CI run has an azure failure.

Previous successful build: https://github.com/apache/arrow/actions/runs/7945396747/job/21692015614

Failure in most recent build: https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

It looks like azurite returned an invalid response.
I've run this multiple times locally but so far have not been able to reproduce.
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
69ed3a5 to
7ed0910CompareFebruary 18, 2024 20:42
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Retrying got the same error in a different test case on a different build https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
@Tom-Newton
Tom-Newton
commented
Feb 18, 2024

Copy link


Copy Markdown

ContributorAuthor
I think #40120 should resolve the Python 3.9 and GCS testbench problem.
I also found an example of the azure tests flaking on mainhttps://github.com/apache/arrow/actions/runs/7915689559/job/21608061673. I'm unsure whether its fair to merge this while this flakiness exists. This PR enables Azure tests in more builds so the flakes will become more frequent.
I created a separate issue to fix this flakiness #40121
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
7ed0910 to
7c27978CompareFebruary 19, 2024 15:01
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/more_azure_builds_and_tests/GH-40079
branch
from
31e4920 to
5e3f846CompareFebruary 19, 2024 19:23
jorisvandenbossche
jorisvandenbossche
reviewed
Feb 20, 2024
View reviewed changes
Comment threadci/conda_env_cpp.txt
Outdated

# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
@jorisvandenbosschejorisvandenbosscheFeb 20, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that is a relevant link. AFAIU there is not really a problem with the 1.11.0 conda package itself. It's that conda did not pin azure-core-cpp in other packages (eg building tiledb with azure-core-cpp against 1.10, and then running that in an env with 1.11 installed, then we got segfaults).
But for the purpose of our CI, we are both building and running against the same azure-core-cpp version, right? In that case, I would expect that building+running with azure-core-cpp would work just fine. Or if not, that means our azurefs implementation might not yet be compatible with the latest azure-core-cpp?
@Tom-NewtonTom-NewtonFeb 20, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I tried using 1.11.0 building from source with cmake and everything worked. With 1.10.0 1.11.0 from conda we got segfaults.
@jorisvandenbosschejorisvandenbosscheFeb 21, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean 1.11 from conda? (not 1.10) Do you remember if there was more information in the logs about the crash?
@Tom-NewtonTom-NewtonFeb 21, 2024

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#40080 (comment) summarises the info we have.
It looks like 1.11.1 is now available from conda though and that works, so I can remove this version restriction.
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Feb 20, 2024
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Feb 21, 2024
@assignUser
assignUser
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
@github-actions crossbow submit -g python
@github-actions
github-actionsBot
commented
Feb 22, 2024

Copy link


Copy Markdown

Revision: 53b9d2c
Submitted crossbow builds: ursacomputing/crossbow @ actions-47658c1ea8
TaskStatus
test-conda-python-3.10GitHub Actions
test-conda-python-3.10-cython2GitHub Actions
test-conda-python-3.10-hdfs-2.9.2GitHub Actions
test-conda-python-3.10-hdfs-3.2.1GitHub Actions
test-conda-python-3.10-pandas-latestGitHub Actions
test-conda-python-3.10-pandas-nightlyGitHub Actions
test-conda-python-3.10-spark-v3.5.0GitHub Actions
test-conda-python-3.10-substraitGitHub Actions
test-conda-python-3.11GitHub Actions
test-conda-python-3.11-dask-latestGitHub Actions
test-conda-python-3.11-dask-upstream_develGitHub Actions
test-conda-python-3.11-hypothesisGitHub Actions
test-conda-python-3.11-pandas-upstream_develGitHub Actions
test-conda-python-3.11-spark-masterGitHub Actions
test-conda-python-3.12GitHub Actions
test-conda-python-3.8GitHub Actions
test-conda-python-3.8-pandas-1.0GitHub Actions
test-conda-python-3.8-spark-v3.5.0GitHub Actions
test-conda-python-3.9GitHub Actions
test-conda-python-3.9-pandas-latestGitHub Actions
test-cuda-pythonGitHub Actions
test-debian-11-python-3Azure
test-fedora-39-python-3Azure
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-python-3GitHub Actions
@Tom-Newton
Tom-Newton
commented
Feb 22, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Remaining CI failures:

s3fs-test https://github.com/apache/arrow/actions/runs/7987315113/job/21809549810?pr=40080#step:7:3930

appveyor timeout https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/49237913/messages

A couple of assertion errors in pandas nightly tests https://github.com/ursacomputing/crossbow/actions/runs/7997817187/job/21842956425#step:6:6409
I don't think any of these are related to my changes.
assignUser
assignUser
approved these changes
Feb 22, 2024
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree CI failures seem unrelated but I'll let @raulcd have the last word on that. Thanks :)
Comment threadci/scripts/cpp_test.sh
@kou
kou
commented
Feb 22, 2024

Copy link


Copy Markdown

Member
Those failures are unrelated because they are happen on main too.
kou
kou
approved these changes
Feb 22, 2024
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit b089c6a
into
apache:mainFeb 22, 2024
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Feb 22, 2024
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Feb 22, 2024
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Feb 23, 2024

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit b089c6a.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
@booxterbooxter
mentioned this pull request
Aug 15, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments
@koukoukou approved these changes
@assignUserassignUserassignUser approved these changes
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[CI][Packaging] Enable Azure in more tests and builds


4 participants

@Tom-Newton@kou@assignUser@jorisvandenbossche