GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

GH-37511: [C++] Implement file reads for Azure filesystem - #38269

Merged
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511
Oct 19, 2023
Merged

GH-37511: [C++] Implement file reads for Azure filesystem#38269
bkietz merged 40 commits into
apache:mainfrom
Tom-Newton:tomnewton/azure_filesystem_reads/GH-37511

Conversation

@Tom-Newton

@Tom-NewtonTom-Newton commented Oct 14, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #37511has been automatically assigned in GitHub to PR creator.

@Tom-Newton
Tom-Newtonforce-pushed the tomnewton/azure_filesystem_reads/GH-37511 branch from a872890 to 98e019cCompareOctober 15, 2023 14:20
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs.cc Outdated
Comment threadcpp/src/arrow/filesystem/azurefs_test.cc

TEST_F(TestAzureFileSystem, OpenInputFileInfoInvalid) {
// TODO: When implemented use ASSERT_OK_AND_ASSIGN(info,
// fs->GetFileInfo(PreexistingContainerPath()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

@Tom-NewtonTom-NewtonOct 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

Comment threadcpp/src/arrow/filesystem/azurefs_test.cc Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 18, 2023
@felipecrv

Copy link
Copy Markdown
Contributor

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
std::shared_ptr<const KeyValueMetadata> actual;
ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
- // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is - // probably not the correct behaviour.
+ // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+ // is probably not the correct behaviour.
ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
}
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

@bkietz
bkietz merged commit 23dfd0e into apache:mainOct 19, 2023
@bkietzbkietz removed the awaiting change review Awaiting change review label Oct 19, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Oct 19, 2023
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…he#38269)
### Rationale for this change
We want a C++ implementation of an Azure filesystem. Reading files is the first step. ### What changes are included in this PR?
Adds an implementation of `io::RandomAccessFile` for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from apache#12914. Using this `io::RandomAccessFile` implementation we implement the input file and stream methods of the `AzureFileSystem`. I've made a few changes to the implementation from apache#12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with `azurite`, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems. ### Are these changes tested?
Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on apache#12914 which recommended this approach. There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented. ### Are there any user-facing changes?
Yes. File reads using the Azure filesystem are now supported. * Closes: apache#37511
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Implement file reads for Azure filesystem

4 participants

@Tom-Newton@felipecrv@bkietz@pitrou