GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988

Merged
kou merged 7 commits into
apache:mainfrom
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged

GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom
Tom-Newton:tomnewton/configure_azurite/GH-36886

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Aug 1, 2023•
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests

What changes are included in this PR?

Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.

Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.

Are these changes tested?

Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.

Are there any user-facing changes?

No

@github-actions

github-actionsBot commented Aug 1, 2023

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/configure_azurite/GH-36886 branch from 8c5060f to d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton changed the title Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions

github-actionsBot commented Aug 1, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newton marked this pull request as ready for review August 4, 2023 16:32
kou
kou reviewed Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>

@koukouAug 7, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use

arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35 in 7cbbd3e

#include<algorithm>// Missing include in boost/process
#defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
#include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
#include<boost/process.hpp>
?

@Tom-NewtonTom-NewtonAug 7, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 7, 2023
kou
kou approved these changes Aug 7, 2023
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@kou
kou merged commit f055f5e into apache:mainAug 7, 2023
@koukou removed the awaiting change review Awaiting change review label Aug 7, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Aug 7, 2023
@koukou mentioned this pull request Aug 11, 2023
Closed
@conbench-apache-arrow

conbench-apache-arrowBot commented Aug 11, 2023

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees

No one assigned

Labels

awaiting mergeAwaiting mergeComponent: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem

2 participants

@Tom-Newton@kou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988
Merged
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Aug 1, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
What changes are included in this PR?
Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.
Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.
Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/configure_azurite/GH-36886
branch
from
8c5060f to
d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton
changed the title
Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
August 4, 2023 16:32
kou
kou
reviewed
Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>
@koukouAug 7, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use 
arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35
in
7cbbd3e
 #include<algorithm>// Missing include in boost/process
 #defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
 #include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
 #include<boost/process.hpp>

?
@Tom-NewtonTom-NewtonAug 7, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Aug 7, 2023
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit f055f5e
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@koukou
mentioned this pull request
Aug 11, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem


2 participants

@Tom-Newton@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988
Merged
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Aug 1, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
What changes are included in this PR?
Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.
Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.
Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/configure_azurite/GH-36886
branch
from
8c5060f to
d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton
changed the title
Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
August 4, 2023 16:32
kou
kou
reviewed
Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>
@koukouAug 7, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use 
arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35
in
7cbbd3e
 #include<algorithm>// Missing include in boost/process
 #defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
 #include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
 #include<boost/process.hpp>

?
@Tom-NewtonTom-NewtonAug 7, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Aug 7, 2023
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit f055f5e
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@koukou
mentioned this pull request
Aug 11, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem


2 participants

@Tom-Newton@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988
Merged
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Aug 1, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
What changes are included in this PR?
Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.
Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.
Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/configure_azurite/GH-36886
branch
from
8c5060f to
d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton
changed the title
Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
August 4, 2023 16:32
kou
kou
reviewed
Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>
@koukouAug 7, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use 
arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35
in
7cbbd3e
 #include<algorithm>// Missing include in boost/process
 #defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
 #include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
 #include<boost/process.hpp>

?
@Tom-NewtonTom-NewtonAug 7, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Aug 7, 2023
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit f055f5e
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@koukou
mentioned this pull request
Aug 11, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem


2 participants

@Tom-Newton@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988
Merged
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Aug 1, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
What changes are included in this PR?
Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.
Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.
Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/configure_azurite/GH-36886
branch
from
8c5060f to
d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton
changed the title
Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
August 4, 2023 16:32
kou
kou
reviewed
Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>
@koukouAug 7, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use 
arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35
in
7cbbd3e
 #include<algorithm>// Missing include in boost/process
 #defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
 #include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
 #include<boost/process.hpp>

?
@Tom-NewtonTom-NewtonAug 7, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Aug 7, 2023
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit f055f5e
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@koukou
mentioned this pull request
Aug 11, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem


2 participants

@Tom-Newton@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988
Merged
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Aug 1, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
What changes are included in this PR?
Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.
Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.
Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/configure_azurite/GH-36886
branch
from
8c5060f to
d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton
changed the title
Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
August 4, 2023 16:32
kou
kou
reviewed
Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>
@koukouAug 7, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use 
arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35
in
7cbbd3e
 #include<algorithm>// Missing include in boost/process
 #defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
 #include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
 #include<boost/process.hpp>

?
@Tom-NewtonTom-NewtonAug 7, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Aug 7, 2023
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit f055f5e
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@koukou
mentioned this pull request
Aug 11, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem


2 participants

@Tom-Newton@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988
Merged
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Aug 1, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
What changes are included in this PR?
Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.
Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.
Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/configure_azurite/GH-36886
branch
from
8c5060f to
d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton
changed the title
Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
August 4, 2023 16:32
kou
kou
reviewed
Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>
@koukouAug 7, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use 
arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35
in
7cbbd3e
 #include<algorithm>// Missing include in boost/process
 #defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
 #include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
 #include<boost/process.hpp>

?
@Tom-NewtonTom-NewtonAug 7, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Aug 7, 2023
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit f055f5e
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@koukou
mentioned this pull request
Aug 11, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem


2 participants

@Tom-Newton@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem - #36988
Merged
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Aug 7, 2023
Merged
GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystem#36988
kou merged 7 commits into
apache:mainfrom 
Tom-Newton:tomnewton/configure_azurite/GH-36886
Conversation
@Tom-Newton
@Tom-NewtonTom-Newton
commented
Aug 1, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Rationale for this change
We need to write tests for #18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
What changes are included in this PR?
Extract the azurite related changes from #12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR.
Currently azurite is configured for all the environments where ARROW_AZURE was enabled by #35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds.
Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this azurite config in #36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this azurite setup PR.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@Tom-Newton
Tom-Newtonforce-pushed
the
tomnewton/configure_azurite/GH-36886
branch
from
8c5060f to
d1102fcCompareAugust 1, 2023 22:53
@Tom-NewtonTom-Newton
changed the title
Tomnewton/configure azurite/gh 36886GH-36886: [C++] Configure azurite in preparation for testing Azure C++ filesystemAug 1, 2023
@github-actions
github-actionsBot
commented
Aug 1, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36886has been automatically assigned in GitHub to PR creator.
@Tom-Newton
Tom-Newton
marked this pull request as ready for review
August 4, 2023 16:32
kou
kou
reviewed
Aug 7, 2023
View reviewed changes
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
// under the License.

#include "arrow/filesystem/azurefs.h"
#include <boost/process.hpp>
@koukouAug 7, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use 
arrow/cpp/src/arrow/filesystem/gcsfs_test.cc

Lines 18 to 35
in
7cbbd3e
 #include<algorithm>// Missing include in boost/process
 #defineBOOST_NO_CXX98_FUNCTION_BASE// ARROW-17805
// This boost/asio/io_context.hpp include is needless for no MinGW
// build.
//
// This is for including boost/asio/detail/socket_types.hpp before any
// "#include <windows.h>". boost/asio/detail/socket_types.hpp doesn't
// work if windows.h is already included. boost/process.h ->
// boost/process/args.hpp -> boost/process/detail/basic_cmd.hpp
// includes windows.h. boost/process/args.hpp is included before
// boost/process/async.h that includes
// boost/asio/detail/socket_types.hpp implicitly is included.
 #include<boost/asio/io_context.hpp>
// We need BOOST_USE_WINDOWS_H definition with MinGW when we use
// boost/process.hpp. See BOOST_USE_WINDOWS_H=1 in
// cpp/cmake_modules/ThirdpartyToolchain.cmake for details.
 #include<boost/process.hpp>

?
@Tom-NewtonTom-NewtonAug 7, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason not to
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Aug 7, 2023
@Tom-Newton@kou
Update cpp/src/arrow/filesystem/azurefs_test.cc
b5a4c04
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Aug 7, 2023
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@kou
kou
merged commit f055f5e
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting change review
Awaiting change review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@koukou
mentioned this pull request
Aug 11, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit f055f5e.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@Tom-Newton@kou@loicalleyne
apacheGH-36886: [C++] Configureazuritein preparation for testing …
541b56f
…Azure C++ filesystem (apache#36988)
### Rationale for this change
We need to write tests for apache#18014. azurite is like a fake Azure blob storage so it can be used to write integration tests
### What changes are included in this PR?
Extract the `azurite` related changes from apache#12914 to create a smaller PR that's easier to review. I have made very minimal changes compared to that PR. Currently `azurite` is configured for all the environments where `ARROW_AZURE` was enabled by apache#35701. I assume its deliberate that its not enabled yet for windows, alpine, conda, debian or fedora builds. ### Are these changes tested?
Its tested by there aren't really any good tests in this PR. I used this `azurite` config in apache#36835 to make an integration test that uses the Azure C++ SDK. On its own we can't really write tests for this `azurite` setup PR. ### Are there any user-facing changes?
No
* Closes: apache#36886
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[C++] Configure azurite in preparation for testing Azure C++ filesystem


2 participants

@Tom-Newton@kou