GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kou merged 9 commits into from
Mar 11, 2024
Merged

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40325

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
704b69e
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…
kou Mar 3, 2024
9e7d75e
Update example URIs
kou Mar 3, 2024
81076bc
Add the document URI for abfs:// and abfss://
kou Mar 3, 2024
fb692de
Remove needless empty container check
kou Mar 5, 2024
376017f
Remove needless options_map
kou Mar 5, 2024
b8e6b83
Implement discussed spec
kou Mar 7, 2024
5927be8
Escape special characters
kou Mar 7, 2024
0a76645
Fix lint error
kou Mar 7, 2024
58d948f
Fix lint error
kou Mar 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions cpp/src/arrow/filesystem/azurefs.cc
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
std::string* out_path) {
const auto host = uri.host();
std::string path;
if (arrow::internal::EndsWith(host, blob_storage_authority)) {
account_name = host.substr(0, host.size() - blob_storage_authority.size());
path = internal::RemoveLeadingSlash(uri.path());
} else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
account_name = host.substr(0, host.size() - dfs_storage_authority.size());
path = internal::ConcatAbstractPath(uri.username(), uri.path());
} else {
account_name = uri.username();
const auto port_text = uri.port_text();
if (host.find(".") == std::string::npos && port_text.empty()) {
// abfs://container/dir/file
path = internal::ConcatAbstractPath(host, uri.path());
} else {
// abfs://host.domain/container/dir/file
// abfs://host.domain:port/container/dir/file
// abfs://host:port/container/dir/file
std::string host_port = host;
if (!port_text.empty()) {
host_port += ":" + port_text;
}
blob_storage_authority = host_port;
dfs_storage_authority = host_port;
path = internal::RemoveLeadingSlash(uri.path());
}
}
if (out_path != nullptr) {
*out_path = path;
}
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
const auto account_key = uri.password();
std::optional<CredentialKind> credential_kind;
std::optional<std::string> credential_kind_value;
std::string tenant_id;
std::string client_id;
std::string client_secret;
ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
for (const auto& kv : options_items) {
if (kv.first == "blob_storage_authority") {
blob_storage_authority = kv.second;
} else if (kv.first == "dfs_storage_authority") {
dfs_storage_authority = kv.second;
} else if (kv.first == "credential_kind") {

@felipecrvfelipecrvMar 4, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

@koukouMar 5, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

@felipecrvfelipecrvMar 5, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

@koukouMar 6, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

@felipecrvfelipecrvMar 6, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

@koukouMar 7, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

@koukouMar 7, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

@felipecrvfelipecrvMar 7, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

if (kv.second == "default") {
credential_kind = CredentialKind::kDefault;
} else if (kv.second == "anonymous") {
credential_kind = CredentialKind::kAnonymous;
} else if (kv.second == "workload_identity") {
credential_kind = CredentialKind::kWorkloadIdentity;
} else {
// Other credential kinds should be inferred from the given
// parameters automatically.
return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
}
credential_kind_value = kv.second;
} else if (kv.first == "tenant_id") {
tenant_id = kv.second;
} else if (kv.first == "client_id") {
client_id = kv.second;
} else if (kv.first == "client_secret") {
client_secret = kv.second;
} else if (kv.first == "enable_tls") {
ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
if (enable_tls) {
blob_storage_scheme = "https";
dfs_storage_scheme = "https";
} else {
blob_storage_scheme = "http";
dfs_storage_scheme = "http";
}
} else {
return Status::Invalid(
"Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
}
}

if (credential_kind) {
if (!account_key.empty()) {
return Status::Invalid("Password must not be specified with credential_kind=",
*credential_kind_value);
}
if (!tenant_id.empty()) {
return Status::Invalid("tenant_id must not be specified with credential_kind=",
*credential_kind_value);
}
if (!client_id.empty()) {
return Status::Invalid("client_id must not be specified with credential_kind=",
*credential_kind_value);
}
if (!client_secret.empty()) {
return Status::Invalid("client_secret must not be specified with credential_kind=",
*credential_kind_value);
}

switch (*credential_kind) {
case CredentialKind::kAnonymous:
RETURN_NOT_OK(ConfigureAnonymousCredential());
break;
case CredentialKind::kWorkloadIdentity:
RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
break;
default:
// Default credential
break;
}
} else {
if (!account_key.empty()) {
// With password
if (!tenant_id.empty()) {
return Status::Invalid("tenant_id must not be specified with password");
}
if (!client_id.empty()) {
return Status::Invalid("client_id must not be specified with password");
}
if (!client_secret.empty()) {
return Status::Invalid("client_secret must not be specified with password");
}
RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
} else {
// Without password
if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
// No related parameters
if (account_name.empty()) {
RETURN_NOT_OK(ConfigureAnonymousCredential());
} else {
// Default credential
}
} else {
// One or more tenant_id, client_id or client_secret are specified
if (client_id.empty()) {
return Status::Invalid("client_id must be specified");
}
if (tenant_id.empty() && client_secret.empty()) {
RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
} else if (!tenant_id.empty() && !client_secret.empty()) {
RETURN_NOT_OK(
ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
} else {
return Status::Invalid("Both of tenant_id and client_secret must be specified");
}
}
}
}
return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
std::string* out_path) {
AzureOptions options;
options.ExtractFromUriSchemeAndHierPart(uri, out_path);
RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
std::string* out_path) {
arrow::internal::Uri uri;
RETURN_NOT_OK(uri.Parse(uri_string));
return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
// TODO(GH-38598): update here when more auth methods are added.
const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down
56 changes: 56 additions & 0 deletions cpp/src/arrow/filesystem/azurefs.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
friend class TestAzureOptions;

/// \brief The name of the Azure Storage Account being accessed.
///
/// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
AzureOptions();
~AzureOptions();

private:
void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
std::string* out_path);
Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

public:
/// \brief Construct a new AzureOptions from an URI.
///
/// Supported formats:
///
/// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
/// [/\<container\>[/\<path\>]]
/// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
/// [/path]
/// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
/// [/\<container\>[/path]]
/// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
///
/// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
/// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
///
/// 3. is for Azure Blob Storage compatible service including Azurite.
///
/// 4. is a shorter version of 1. and 2.
///
/// Note that there is no difference between abfs and abfss. HTTPS is
/// used with abfs by default. You can force to use HTTP by specifying
/// "enable_tls=false" query.
///
/// Supported query parameters:
///
/// * blob_storage_authority: Set AzureOptions::blob_storage_authority
/// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
/// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
/// * credential_kind: One of "default", "anonymous",
/// "workload_identity". If "default" is specified, it's just
/// ignored. If "anonymous" is specified,
/// AzureOptions::ConfigureAnonymousCredential() is called. If
/// "workload_identity" is specified,
/// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
/// * tenant_id: You must specify "client_id" and "client_secret"
/// too. AzureOptions::ConfigureClientSecretCredential() is called.
/// * client_id: If you don't specify "tenant_id" and
/// "client_secret",
/// AzureOptions::ConfigureManagedIdentityCredential() is
/// called. If you specify "tenant_id" and "client_secret" too,
/// AzureOptions::ConfigureClientSecretCredential() is called.
/// * client_secret: You must specify "tenant_id" and "client_id"
/// too. AzureOptions::ConfigureClientSecretCredential() is called.
static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
std::string* out_path);
static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

Status ConfigureDefaultCredential();
Status ConfigureAnonymousCredential();
Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 9 commits into

from

Mar 11, 2024
 Merged


GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()
#40325
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
704b69e

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…


kou Mar 3, 2024
9e7d75e

Update example URIs


kou Mar 3, 2024
81076bc

Add the document URI for abfs:// and abfss://


kou Mar 3, 2024
fb692de

Remove needless empty container check


kou Mar 5, 2024
376017f

Remove needless options_map


kou Mar 5, 2024
b8e6b83

Implement discussed spec


kou Mar 7, 2024
5927be8

Escape special characters


kou Mar 7, 2024
0a76645

Fix lint error


kou Mar 7, 2024
58d948f

Fix lint error


kou Mar 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


166 changes: 166 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path) {
 const auto host = uri.host();
 std::string path;
 if (arrow::internal::EndsWith(host, blob_storage_authority)) {
 account_name = host.substr(0, host.size() - blob_storage_authority.size());
 path = internal::RemoveLeadingSlash(uri.path());
 } else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
 account_name = host.substr(0, host.size() - dfs_storage_authority.size());
 path = internal::ConcatAbstractPath(uri.username(), uri.path());
 } else {
 account_name = uri.username();
 const auto port_text = uri.port_text();
 if (host.find(".") == std::string::npos && port_text.empty()) {
 // abfs://container/dir/file
 path = internal::ConcatAbstractPath(host, uri.path());
 } else {
 // abfs://host.domain/container/dir/file
 // abfs://host.domain:port/container/dir/file
 // abfs://host:port/container/dir/file
 std::string host_port = host;
 if (!port_text.empty()) {
 host_port += ":" + port_text;
 }
 blob_storage_authority = host_port;
 dfs_storage_authority = host_port;
 path = internal::RemoveLeadingSlash(uri.path());
 }
 }
 if (out_path != nullptr) {
 *out_path = path;
 }
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
 const auto account_key = uri.password();
 std::optional<CredentialKind> credential_kind;
 std::optional<std::string> credential_kind_value;
 std::string tenant_id;
 std::string client_id;
 std::string client_secret;
 ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
 for (const auto& kv : options_items) {
 if (kv.first == "blob_storage_authority") {
 blob_storage_authority = kv.second;
 } else if (kv.first == "dfs_storage_authority") {
 dfs_storage_authority = kv.second;
 } else if (kv.first == "credential_kind") {
@felipecrvfelipecrvMar 4, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.
@koukouMar 5, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?
@felipecrvfelipecrvMar 5, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.
@koukouMar 6, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.
@felipecrvfelipecrvMar 6, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:
  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)
How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.
@felipecrvfelipecrvMar 7, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
Sure. That looks good.
 if (kv.second == "default") {
 credential_kind = CredentialKind::kDefault;
 } else if (kv.second == "anonymous") {
 credential_kind = CredentialKind::kAnonymous;
 } else if (kv.second == "workload_identity") {
 credential_kind = CredentialKind::kWorkloadIdentity;
 } else {
 // Other credential kinds should be inferred from the given
 // parameters automatically.
 return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
 }
 credential_kind_value = kv.second;
 } else if (kv.first == "tenant_id") {
 tenant_id = kv.second;
 } else if (kv.first == "client_id") {
 client_id = kv.second;
 } else if (kv.first == "client_secret") {
 client_secret = kv.second;
 } else if (kv.first == "enable_tls") {
 ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
 if (enable_tls) {
 blob_storage_scheme = "https";
 dfs_storage_scheme = "https";
 } else {
 blob_storage_scheme = "http";
 dfs_storage_scheme = "http";
 }
 } else {
 return Status::Invalid(
 "Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
 }
 }

 if (credential_kind) {
 if (!account_key.empty()) {
 return Status::Invalid("Password must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with credential_kind=",
 *credential_kind_value);
 }

 switch (*credential_kind) {
 case CredentialKind::kAnonymous:
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 break;
 case CredentialKind::kWorkloadIdentity:
 RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
 break;
 default:
 // Default credential
 break;
 }
 } else {
 if (!account_key.empty()) {
 // With password
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with password");
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with password");
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with password");
 }
 RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
 } else {
 // Without password
 if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
 // No related parameters
 if (account_name.empty()) {
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 } else {
 // Default credential
 }
 } else {
 // One or more tenant_id, client_id or client_secret are specified
 if (client_id.empty()) {
 return Status::Invalid("client_id must be specified");
 }
 if (tenant_id.empty() && client_secret.empty()) {
 RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
 } else if (!tenant_id.empty() && !client_secret.empty()) {
 RETURN_NOT_OK(
 ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
 } else {
 return Status::Invalid("Both of tenant_id and client_secret must be specified");
 }
 }
 }
 }
 return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
 std::string* out_path) {
 AzureOptions options;
 options.ExtractFromUriSchemeAndHierPart(uri, out_path);
 RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
 return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
 std::string* out_path) {
 arrow::internal::Uri uri;
 RETURN_NOT_OK(uri.Parse(uri_string));
 return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
 // TODO(GH-38598): update here when more auth methods are added.
 const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down

56 changes: 56 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
 friend class TestAzureOptions;

 /// \brief The name of the Azure Storage Account being accessed.
 ///
 /// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
 AzureOptions();
 ~AzureOptions();

 private:
 void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path);
 Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

 public:
 /// \brief Construct a new AzureOptions from an URI.
 ///
 /// Supported formats:
 ///
 /// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
 /// [/\<container\>[/\<path\>]]
 /// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
 /// [/path]
 /// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
 /// [/\<container\>[/path]]
 /// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
 ///
 /// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
 /// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
 ///
 /// 3. is for Azure Blob Storage compatible service including Azurite.
 ///
 /// 4. is a shorter version of 1. and 2.
 ///
 /// Note that there is no difference between abfs and abfss. HTTPS is
 /// used with abfs by default. You can force to use HTTP by specifying
 /// "enable_tls=false" query.
 ///
 /// Supported query parameters:
 ///
 /// * blob_storage_authority: Set AzureOptions::blob_storage_authority
 /// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
 /// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
 /// * credential_kind: One of "default", "anonymous",
 /// "workload_identity". If "default" is specified, it's just
 /// ignored. If "anonymous" is specified,
 /// AzureOptions::ConfigureAnonymousCredential() is called. If
 /// "workload_identity" is specified,
 /// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
 /// * tenant_id: You must specify "client_id" and "client_secret"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_id: If you don't specify "tenant_id" and
 /// "client_secret",
 /// AzureOptions::ConfigureManagedIdentityCredential() is
 /// called. If you specify "tenant_id" and "client_secret" too,
 /// AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_secret: You must specify "tenant_id" and "client_id"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
 std::string* out_path);
 static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

 Status ConfigureDefaultCredential();
 Status ConfigureAnonymousCredential();
 Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 9 commits into

from

Mar 11, 2024
 Merged


GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()
#40325
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
704b69e

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…


kou Mar 3, 2024
9e7d75e

Update example URIs


kou Mar 3, 2024
81076bc

Add the document URI for abfs:// and abfss://


kou Mar 3, 2024
fb692de

Remove needless empty container check


kou Mar 5, 2024
376017f

Remove needless options_map


kou Mar 5, 2024
b8e6b83

Implement discussed spec


kou Mar 7, 2024
5927be8

Escape special characters


kou Mar 7, 2024
0a76645

Fix lint error


kou Mar 7, 2024
58d948f

Fix lint error


kou Mar 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


166 changes: 166 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path) {
 const auto host = uri.host();
 std::string path;
 if (arrow::internal::EndsWith(host, blob_storage_authority)) {
 account_name = host.substr(0, host.size() - blob_storage_authority.size());
 path = internal::RemoveLeadingSlash(uri.path());
 } else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
 account_name = host.substr(0, host.size() - dfs_storage_authority.size());
 path = internal::ConcatAbstractPath(uri.username(), uri.path());
 } else {
 account_name = uri.username();
 const auto port_text = uri.port_text();
 if (host.find(".") == std::string::npos && port_text.empty()) {
 // abfs://container/dir/file
 path = internal::ConcatAbstractPath(host, uri.path());
 } else {
 // abfs://host.domain/container/dir/file
 // abfs://host.domain:port/container/dir/file
 // abfs://host:port/container/dir/file
 std::string host_port = host;
 if (!port_text.empty()) {
 host_port += ":" + port_text;
 }
 blob_storage_authority = host_port;
 dfs_storage_authority = host_port;
 path = internal::RemoveLeadingSlash(uri.path());
 }
 }
 if (out_path != nullptr) {
 *out_path = path;
 }
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
 const auto account_key = uri.password();
 std::optional<CredentialKind> credential_kind;
 std::optional<std::string> credential_kind_value;
 std::string tenant_id;
 std::string client_id;
 std::string client_secret;
 ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
 for (const auto& kv : options_items) {
 if (kv.first == "blob_storage_authority") {
 blob_storage_authority = kv.second;
 } else if (kv.first == "dfs_storage_authority") {
 dfs_storage_authority = kv.second;
 } else if (kv.first == "credential_kind") {
@felipecrvfelipecrvMar 4, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.
@koukouMar 5, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?
@felipecrvfelipecrvMar 5, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.
@koukouMar 6, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.
@felipecrvfelipecrvMar 6, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:
  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)
How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.
@felipecrvfelipecrvMar 7, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
Sure. That looks good.
 if (kv.second == "default") {
 credential_kind = CredentialKind::kDefault;
 } else if (kv.second == "anonymous") {
 credential_kind = CredentialKind::kAnonymous;
 } else if (kv.second == "workload_identity") {
 credential_kind = CredentialKind::kWorkloadIdentity;
 } else {
 // Other credential kinds should be inferred from the given
 // parameters automatically.
 return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
 }
 credential_kind_value = kv.second;
 } else if (kv.first == "tenant_id") {
 tenant_id = kv.second;
 } else if (kv.first == "client_id") {
 client_id = kv.second;
 } else if (kv.first == "client_secret") {
 client_secret = kv.second;
 } else if (kv.first == "enable_tls") {
 ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
 if (enable_tls) {
 blob_storage_scheme = "https";
 dfs_storage_scheme = "https";
 } else {
 blob_storage_scheme = "http";
 dfs_storage_scheme = "http";
 }
 } else {
 return Status::Invalid(
 "Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
 }
 }

 if (credential_kind) {
 if (!account_key.empty()) {
 return Status::Invalid("Password must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with credential_kind=",
 *credential_kind_value);
 }

 switch (*credential_kind) {
 case CredentialKind::kAnonymous:
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 break;
 case CredentialKind::kWorkloadIdentity:
 RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
 break;
 default:
 // Default credential
 break;
 }
 } else {
 if (!account_key.empty()) {
 // With password
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with password");
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with password");
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with password");
 }
 RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
 } else {
 // Without password
 if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
 // No related parameters
 if (account_name.empty()) {
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 } else {
 // Default credential
 }
 } else {
 // One or more tenant_id, client_id or client_secret are specified
 if (client_id.empty()) {
 return Status::Invalid("client_id must be specified");
 }
 if (tenant_id.empty() && client_secret.empty()) {
 RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
 } else if (!tenant_id.empty() && !client_secret.empty()) {
 RETURN_NOT_OK(
 ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
 } else {
 return Status::Invalid("Both of tenant_id and client_secret must be specified");
 }
 }
 }
 }
 return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
 std::string* out_path) {
 AzureOptions options;
 options.ExtractFromUriSchemeAndHierPart(uri, out_path);
 RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
 return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
 std::string* out_path) {
 arrow::internal::Uri uri;
 RETURN_NOT_OK(uri.Parse(uri_string));
 return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
 // TODO(GH-38598): update here when more auth methods are added.
 const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down

56 changes: 56 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
 friend class TestAzureOptions;

 /// \brief The name of the Azure Storage Account being accessed.
 ///
 /// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
 AzureOptions();
 ~AzureOptions();

 private:
 void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path);
 Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

 public:
 /// \brief Construct a new AzureOptions from an URI.
 ///
 /// Supported formats:
 ///
 /// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
 /// [/\<container\>[/\<path\>]]
 /// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
 /// [/path]
 /// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
 /// [/\<container\>[/path]]
 /// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
 ///
 /// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
 /// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
 ///
 /// 3. is for Azure Blob Storage compatible service including Azurite.
 ///
 /// 4. is a shorter version of 1. and 2.
 ///
 /// Note that there is no difference between abfs and abfss. HTTPS is
 /// used with abfs by default. You can force to use HTTP by specifying
 /// "enable_tls=false" query.
 ///
 /// Supported query parameters:
 ///
 /// * blob_storage_authority: Set AzureOptions::blob_storage_authority
 /// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
 /// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
 /// * credential_kind: One of "default", "anonymous",
 /// "workload_identity". If "default" is specified, it's just
 /// ignored. If "anonymous" is specified,
 /// AzureOptions::ConfigureAnonymousCredential() is called. If
 /// "workload_identity" is specified,
 /// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
 /// * tenant_id: You must specify "client_id" and "client_secret"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_id: If you don't specify "tenant_id" and
 /// "client_secret",
 /// AzureOptions::ConfigureManagedIdentityCredential() is
 /// called. If you specify "tenant_id" and "client_secret" too,
 /// AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_secret: You must specify "tenant_id" and "client_id"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
 std::string* out_path);
 static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

 Status ConfigureDefaultCredential();
 Status ConfigureAnonymousCredential();
 Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 9 commits into

from

Mar 11, 2024
 Merged


GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()
#40325
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
704b69e

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…


kou Mar 3, 2024
9e7d75e

Update example URIs


kou Mar 3, 2024
81076bc

Add the document URI for abfs:// and abfss://


kou Mar 3, 2024
fb692de

Remove needless empty container check


kou Mar 5, 2024
376017f

Remove needless options_map


kou Mar 5, 2024
b8e6b83

Implement discussed spec


kou Mar 7, 2024
5927be8

Escape special characters


kou Mar 7, 2024
0a76645

Fix lint error


kou Mar 7, 2024
58d948f

Fix lint error


kou Mar 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


166 changes: 166 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path) {
 const auto host = uri.host();
 std::string path;
 if (arrow::internal::EndsWith(host, blob_storage_authority)) {
 account_name = host.substr(0, host.size() - blob_storage_authority.size());
 path = internal::RemoveLeadingSlash(uri.path());
 } else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
 account_name = host.substr(0, host.size() - dfs_storage_authority.size());
 path = internal::ConcatAbstractPath(uri.username(), uri.path());
 } else {
 account_name = uri.username();
 const auto port_text = uri.port_text();
 if (host.find(".") == std::string::npos && port_text.empty()) {
 // abfs://container/dir/file
 path = internal::ConcatAbstractPath(host, uri.path());
 } else {
 // abfs://host.domain/container/dir/file
 // abfs://host.domain:port/container/dir/file
 // abfs://host:port/container/dir/file
 std::string host_port = host;
 if (!port_text.empty()) {
 host_port += ":" + port_text;
 }
 blob_storage_authority = host_port;
 dfs_storage_authority = host_port;
 path = internal::RemoveLeadingSlash(uri.path());
 }
 }
 if (out_path != nullptr) {
 *out_path = path;
 }
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
 const auto account_key = uri.password();
 std::optional<CredentialKind> credential_kind;
 std::optional<std::string> credential_kind_value;
 std::string tenant_id;
 std::string client_id;
 std::string client_secret;
 ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
 for (const auto& kv : options_items) {
 if (kv.first == "blob_storage_authority") {
 blob_storage_authority = kv.second;
 } else if (kv.first == "dfs_storage_authority") {
 dfs_storage_authority = kv.second;
 } else if (kv.first == "credential_kind") {
@felipecrvfelipecrvMar 4, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.
@koukouMar 5, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?
@felipecrvfelipecrvMar 5, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.
@koukouMar 6, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.
@felipecrvfelipecrvMar 6, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:
  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)
How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.
@felipecrvfelipecrvMar 7, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
Sure. That looks good.
 if (kv.second == "default") {
 credential_kind = CredentialKind::kDefault;
 } else if (kv.second == "anonymous") {
 credential_kind = CredentialKind::kAnonymous;
 } else if (kv.second == "workload_identity") {
 credential_kind = CredentialKind::kWorkloadIdentity;
 } else {
 // Other credential kinds should be inferred from the given
 // parameters automatically.
 return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
 }
 credential_kind_value = kv.second;
 } else if (kv.first == "tenant_id") {
 tenant_id = kv.second;
 } else if (kv.first == "client_id") {
 client_id = kv.second;
 } else if (kv.first == "client_secret") {
 client_secret = kv.second;
 } else if (kv.first == "enable_tls") {
 ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
 if (enable_tls) {
 blob_storage_scheme = "https";
 dfs_storage_scheme = "https";
 } else {
 blob_storage_scheme = "http";
 dfs_storage_scheme = "http";
 }
 } else {
 return Status::Invalid(
 "Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
 }
 }

 if (credential_kind) {
 if (!account_key.empty()) {
 return Status::Invalid("Password must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with credential_kind=",
 *credential_kind_value);
 }

 switch (*credential_kind) {
 case CredentialKind::kAnonymous:
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 break;
 case CredentialKind::kWorkloadIdentity:
 RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
 break;
 default:
 // Default credential
 break;
 }
 } else {
 if (!account_key.empty()) {
 // With password
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with password");
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with password");
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with password");
 }
 RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
 } else {
 // Without password
 if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
 // No related parameters
 if (account_name.empty()) {
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 } else {
 // Default credential
 }
 } else {
 // One or more tenant_id, client_id or client_secret are specified
 if (client_id.empty()) {
 return Status::Invalid("client_id must be specified");
 }
 if (tenant_id.empty() && client_secret.empty()) {
 RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
 } else if (!tenant_id.empty() && !client_secret.empty()) {
 RETURN_NOT_OK(
 ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
 } else {
 return Status::Invalid("Both of tenant_id and client_secret must be specified");
 }
 }
 }
 }
 return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
 std::string* out_path) {
 AzureOptions options;
 options.ExtractFromUriSchemeAndHierPart(uri, out_path);
 RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
 return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
 std::string* out_path) {
 arrow::internal::Uri uri;
 RETURN_NOT_OK(uri.Parse(uri_string));
 return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
 // TODO(GH-38598): update here when more auth methods are added.
 const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down

56 changes: 56 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
 friend class TestAzureOptions;

 /// \brief The name of the Azure Storage Account being accessed.
 ///
 /// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
 AzureOptions();
 ~AzureOptions();

 private:
 void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path);
 Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

 public:
 /// \brief Construct a new AzureOptions from an URI.
 ///
 /// Supported formats:
 ///
 /// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
 /// [/\<container\>[/\<path\>]]
 /// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
 /// [/path]
 /// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
 /// [/\<container\>[/path]]
 /// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
 ///
 /// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
 /// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
 ///
 /// 3. is for Azure Blob Storage compatible service including Azurite.
 ///
 /// 4. is a shorter version of 1. and 2.
 ///
 /// Note that there is no difference between abfs and abfss. HTTPS is
 /// used with abfs by default. You can force to use HTTP by specifying
 /// "enable_tls=false" query.
 ///
 /// Supported query parameters:
 ///
 /// * blob_storage_authority: Set AzureOptions::blob_storage_authority
 /// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
 /// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
 /// * credential_kind: One of "default", "anonymous",
 /// "workload_identity". If "default" is specified, it's just
 /// ignored. If "anonymous" is specified,
 /// AzureOptions::ConfigureAnonymousCredential() is called. If
 /// "workload_identity" is specified,
 /// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
 /// * tenant_id: You must specify "client_id" and "client_secret"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_id: If you don't specify "tenant_id" and
 /// "client_secret",
 /// AzureOptions::ConfigureManagedIdentityCredential() is
 /// called. If you specify "tenant_id" and "client_secret" too,
 /// AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_secret: You must specify "tenant_id" and "client_id"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
 std::string* out_path);
 static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

 Status ConfigureDefaultCredential();
 Status ConfigureAnonymousCredential();
 Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 9 commits into

from

Mar 11, 2024
 Merged


GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()
#40325
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
704b69e

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…


kou Mar 3, 2024
9e7d75e

Update example URIs


kou Mar 3, 2024
81076bc

Add the document URI for abfs:// and abfss://


kou Mar 3, 2024
fb692de

Remove needless empty container check


kou Mar 5, 2024
376017f

Remove needless options_map


kou Mar 5, 2024
b8e6b83

Implement discussed spec


kou Mar 7, 2024
5927be8

Escape special characters


kou Mar 7, 2024
0a76645

Fix lint error


kou Mar 7, 2024
58d948f

Fix lint error


kou Mar 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


166 changes: 166 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path) {
 const auto host = uri.host();
 std::string path;
 if (arrow::internal::EndsWith(host, blob_storage_authority)) {
 account_name = host.substr(0, host.size() - blob_storage_authority.size());
 path = internal::RemoveLeadingSlash(uri.path());
 } else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
 account_name = host.substr(0, host.size() - dfs_storage_authority.size());
 path = internal::ConcatAbstractPath(uri.username(), uri.path());
 } else {
 account_name = uri.username();
 const auto port_text = uri.port_text();
 if (host.find(".") == std::string::npos && port_text.empty()) {
 // abfs://container/dir/file
 path = internal::ConcatAbstractPath(host, uri.path());
 } else {
 // abfs://host.domain/container/dir/file
 // abfs://host.domain:port/container/dir/file
 // abfs://host:port/container/dir/file
 std::string host_port = host;
 if (!port_text.empty()) {
 host_port += ":" + port_text;
 }
 blob_storage_authority = host_port;
 dfs_storage_authority = host_port;
 path = internal::RemoveLeadingSlash(uri.path());
 }
 }
 if (out_path != nullptr) {
 *out_path = path;
 }
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
 const auto account_key = uri.password();
 std::optional<CredentialKind> credential_kind;
 std::optional<std::string> credential_kind_value;
 std::string tenant_id;
 std::string client_id;
 std::string client_secret;
 ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
 for (const auto& kv : options_items) {
 if (kv.first == "blob_storage_authority") {
 blob_storage_authority = kv.second;
 } else if (kv.first == "dfs_storage_authority") {
 dfs_storage_authority = kv.second;
 } else if (kv.first == "credential_kind") {
@felipecrvfelipecrvMar 4, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.
@koukouMar 5, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?
@felipecrvfelipecrvMar 5, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.
@koukouMar 6, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.
@felipecrvfelipecrvMar 6, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:
  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)
How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.
@felipecrvfelipecrvMar 7, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
Sure. That looks good.
 if (kv.second == "default") {
 credential_kind = CredentialKind::kDefault;
 } else if (kv.second == "anonymous") {
 credential_kind = CredentialKind::kAnonymous;
 } else if (kv.second == "workload_identity") {
 credential_kind = CredentialKind::kWorkloadIdentity;
 } else {
 // Other credential kinds should be inferred from the given
 // parameters automatically.
 return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
 }
 credential_kind_value = kv.second;
 } else if (kv.first == "tenant_id") {
 tenant_id = kv.second;
 } else if (kv.first == "client_id") {
 client_id = kv.second;
 } else if (kv.first == "client_secret") {
 client_secret = kv.second;
 } else if (kv.first == "enable_tls") {
 ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
 if (enable_tls) {
 blob_storage_scheme = "https";
 dfs_storage_scheme = "https";
 } else {
 blob_storage_scheme = "http";
 dfs_storage_scheme = "http";
 }
 } else {
 return Status::Invalid(
 "Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
 }
 }

 if (credential_kind) {
 if (!account_key.empty()) {
 return Status::Invalid("Password must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with credential_kind=",
 *credential_kind_value);
 }

 switch (*credential_kind) {
 case CredentialKind::kAnonymous:
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 break;
 case CredentialKind::kWorkloadIdentity:
 RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
 break;
 default:
 // Default credential
 break;
 }
 } else {
 if (!account_key.empty()) {
 // With password
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with password");
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with password");
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with password");
 }
 RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
 } else {
 // Without password
 if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
 // No related parameters
 if (account_name.empty()) {
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 } else {
 // Default credential
 }
 } else {
 // One or more tenant_id, client_id or client_secret are specified
 if (client_id.empty()) {
 return Status::Invalid("client_id must be specified");
 }
 if (tenant_id.empty() && client_secret.empty()) {
 RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
 } else if (!tenant_id.empty() && !client_secret.empty()) {
 RETURN_NOT_OK(
 ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
 } else {
 return Status::Invalid("Both of tenant_id and client_secret must be specified");
 }
 }
 }
 }
 return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
 std::string* out_path) {
 AzureOptions options;
 options.ExtractFromUriSchemeAndHierPart(uri, out_path);
 RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
 return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
 std::string* out_path) {
 arrow::internal::Uri uri;
 RETURN_NOT_OK(uri.Parse(uri_string));
 return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
 // TODO(GH-38598): update here when more auth methods are added.
 const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down

56 changes: 56 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
 friend class TestAzureOptions;

 /// \brief The name of the Azure Storage Account being accessed.
 ///
 /// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
 AzureOptions();
 ~AzureOptions();

 private:
 void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path);
 Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

 public:
 /// \brief Construct a new AzureOptions from an URI.
 ///
 /// Supported formats:
 ///
 /// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
 /// [/\<container\>[/\<path\>]]
 /// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
 /// [/path]
 /// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
 /// [/\<container\>[/path]]
 /// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
 ///
 /// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
 /// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
 ///
 /// 3. is for Azure Blob Storage compatible service including Azurite.
 ///
 /// 4. is a shorter version of 1. and 2.
 ///
 /// Note that there is no difference between abfs and abfss. HTTPS is
 /// used with abfs by default. You can force to use HTTP by specifying
 /// "enable_tls=false" query.
 ///
 /// Supported query parameters:
 ///
 /// * blob_storage_authority: Set AzureOptions::blob_storage_authority
 /// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
 /// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
 /// * credential_kind: One of "default", "anonymous",
 /// "workload_identity". If "default" is specified, it's just
 /// ignored. If "anonymous" is specified,
 /// AzureOptions::ConfigureAnonymousCredential() is called. If
 /// "workload_identity" is specified,
 /// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
 /// * tenant_id: You must specify "client_id" and "client_secret"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_id: If you don't specify "tenant_id" and
 /// "client_secret",
 /// AzureOptions::ConfigureManagedIdentityCredential() is
 /// called. If you specify "tenant_id" and "client_secret" too,
 /// AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_secret: You must specify "tenant_id" and "client_id"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
 std::string* out_path);
 static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

 Status ConfigureDefaultCredential();
 Status ConfigureAnonymousCredential();
 Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 9 commits into

from

Mar 11, 2024
 Merged


GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()
#40325
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
704b69e

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…


kou Mar 3, 2024
9e7d75e

Update example URIs


kou Mar 3, 2024
81076bc

Add the document URI for abfs:// and abfss://


kou Mar 3, 2024
fb692de

Remove needless empty container check


kou Mar 5, 2024
376017f

Remove needless options_map


kou Mar 5, 2024
b8e6b83

Implement discussed spec


kou Mar 7, 2024
5927be8

Escape special characters


kou Mar 7, 2024
0a76645

Fix lint error


kou Mar 7, 2024
58d948f

Fix lint error


kou Mar 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


166 changes: 166 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path) {
 const auto host = uri.host();
 std::string path;
 if (arrow::internal::EndsWith(host, blob_storage_authority)) {
 account_name = host.substr(0, host.size() - blob_storage_authority.size());
 path = internal::RemoveLeadingSlash(uri.path());
 } else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
 account_name = host.substr(0, host.size() - dfs_storage_authority.size());
 path = internal::ConcatAbstractPath(uri.username(), uri.path());
 } else {
 account_name = uri.username();
 const auto port_text = uri.port_text();
 if (host.find(".") == std::string::npos && port_text.empty()) {
 // abfs://container/dir/file
 path = internal::ConcatAbstractPath(host, uri.path());
 } else {
 // abfs://host.domain/container/dir/file
 // abfs://host.domain:port/container/dir/file
 // abfs://host:port/container/dir/file
 std::string host_port = host;
 if (!port_text.empty()) {
 host_port += ":" + port_text;
 }
 blob_storage_authority = host_port;
 dfs_storage_authority = host_port;
 path = internal::RemoveLeadingSlash(uri.path());
 }
 }
 if (out_path != nullptr) {
 *out_path = path;
 }
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
 const auto account_key = uri.password();
 std::optional<CredentialKind> credential_kind;
 std::optional<std::string> credential_kind_value;
 std::string tenant_id;
 std::string client_id;
 std::string client_secret;
 ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
 for (const auto& kv : options_items) {
 if (kv.first == "blob_storage_authority") {
 blob_storage_authority = kv.second;
 } else if (kv.first == "dfs_storage_authority") {
 dfs_storage_authority = kv.second;
 } else if (kv.first == "credential_kind") {
@felipecrvfelipecrvMar 4, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.
@koukouMar 5, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?
@felipecrvfelipecrvMar 5, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.
@koukouMar 6, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.
@felipecrvfelipecrvMar 6, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:
  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)
How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.
@felipecrvfelipecrvMar 7, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
Sure. That looks good.
 if (kv.second == "default") {
 credential_kind = CredentialKind::kDefault;
 } else if (kv.second == "anonymous") {
 credential_kind = CredentialKind::kAnonymous;
 } else if (kv.second == "workload_identity") {
 credential_kind = CredentialKind::kWorkloadIdentity;
 } else {
 // Other credential kinds should be inferred from the given
 // parameters automatically.
 return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
 }
 credential_kind_value = kv.second;
 } else if (kv.first == "tenant_id") {
 tenant_id = kv.second;
 } else if (kv.first == "client_id") {
 client_id = kv.second;
 } else if (kv.first == "client_secret") {
 client_secret = kv.second;
 } else if (kv.first == "enable_tls") {
 ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
 if (enable_tls) {
 blob_storage_scheme = "https";
 dfs_storage_scheme = "https";
 } else {
 blob_storage_scheme = "http";
 dfs_storage_scheme = "http";
 }
 } else {
 return Status::Invalid(
 "Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
 }
 }

 if (credential_kind) {
 if (!account_key.empty()) {
 return Status::Invalid("Password must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with credential_kind=",
 *credential_kind_value);
 }

 switch (*credential_kind) {
 case CredentialKind::kAnonymous:
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 break;
 case CredentialKind::kWorkloadIdentity:
 RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
 break;
 default:
 // Default credential
 break;
 }
 } else {
 if (!account_key.empty()) {
 // With password
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with password");
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with password");
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with password");
 }
 RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
 } else {
 // Without password
 if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
 // No related parameters
 if (account_name.empty()) {
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 } else {
 // Default credential
 }
 } else {
 // One or more tenant_id, client_id or client_secret are specified
 if (client_id.empty()) {
 return Status::Invalid("client_id must be specified");
 }
 if (tenant_id.empty() && client_secret.empty()) {
 RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
 } else if (!tenant_id.empty() && !client_secret.empty()) {
 RETURN_NOT_OK(
 ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
 } else {
 return Status::Invalid("Both of tenant_id and client_secret must be specified");
 }
 }
 }
 }
 return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
 std::string* out_path) {
 AzureOptions options;
 options.ExtractFromUriSchemeAndHierPart(uri, out_path);
 RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
 return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
 std::string* out_path) {
 arrow::internal::Uri uri;
 RETURN_NOT_OK(uri.Parse(uri_string));
 return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
 // TODO(GH-38598): update here when more auth methods are added.
 const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down

56 changes: 56 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
 friend class TestAzureOptions;

 /// \brief The name of the Azure Storage Account being accessed.
 ///
 /// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
 AzureOptions();
 ~AzureOptions();

 private:
 void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path);
 Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

 public:
 /// \brief Construct a new AzureOptions from an URI.
 ///
 /// Supported formats:
 ///
 /// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
 /// [/\<container\>[/\<path\>]]
 /// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
 /// [/path]
 /// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
 /// [/\<container\>[/path]]
 /// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
 ///
 /// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
 /// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
 ///
 /// 3. is for Azure Blob Storage compatible service including Azurite.
 ///
 /// 4. is a shorter version of 1. and 2.
 ///
 /// Note that there is no difference between abfs and abfss. HTTPS is
 /// used with abfs by default. You can force to use HTTP by specifying
 /// "enable_tls=false" query.
 ///
 /// Supported query parameters:
 ///
 /// * blob_storage_authority: Set AzureOptions::blob_storage_authority
 /// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
 /// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
 /// * credential_kind: One of "default", "anonymous",
 /// "workload_identity". If "default" is specified, it's just
 /// ignored. If "anonymous" is specified,
 /// AzureOptions::ConfigureAnonymousCredential() is called. If
 /// "workload_identity" is specified,
 /// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
 /// * tenant_id: You must specify "client_id" and "client_secret"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_id: If you don't specify "tenant_id" and
 /// "client_secret",
 /// AzureOptions::ConfigureManagedIdentityCredential() is
 /// called. If you specify "tenant_id" and "client_secret" too,
 /// AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_secret: You must specify "tenant_id" and "client_id"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
 std::string* out_path);
 static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

 Status ConfigureDefaultCredential();
 Status ConfigureAnonymousCredential();
 Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 9 commits into

from

Mar 11, 2024
 Merged


GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()
#40325
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
704b69e

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…


kou Mar 3, 2024
9e7d75e

Update example URIs


kou Mar 3, 2024
81076bc

Add the document URI for abfs:// and abfss://


kou Mar 3, 2024
fb692de

Remove needless empty container check


kou Mar 5, 2024
376017f

Remove needless options_map


kou Mar 5, 2024
b8e6b83

Implement discussed spec


kou Mar 7, 2024
5927be8

Escape special characters


kou Mar 7, 2024
0a76645

Fix lint error


kou Mar 7, 2024
58d948f

Fix lint error


kou Mar 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


166 changes: 166 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path) {
 const auto host = uri.host();
 std::string path;
 if (arrow::internal::EndsWith(host, blob_storage_authority)) {
 account_name = host.substr(0, host.size() - blob_storage_authority.size());
 path = internal::RemoveLeadingSlash(uri.path());
 } else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
 account_name = host.substr(0, host.size() - dfs_storage_authority.size());
 path = internal::ConcatAbstractPath(uri.username(), uri.path());
 } else {
 account_name = uri.username();
 const auto port_text = uri.port_text();
 if (host.find(".") == std::string::npos && port_text.empty()) {
 // abfs://container/dir/file
 path = internal::ConcatAbstractPath(host, uri.path());
 } else {
 // abfs://host.domain/container/dir/file
 // abfs://host.domain:port/container/dir/file
 // abfs://host:port/container/dir/file
 std::string host_port = host;
 if (!port_text.empty()) {
 host_port += ":" + port_text;
 }
 blob_storage_authority = host_port;
 dfs_storage_authority = host_port;
 path = internal::RemoveLeadingSlash(uri.path());
 }
 }
 if (out_path != nullptr) {
 *out_path = path;
 }
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
 const auto account_key = uri.password();
 std::optional<CredentialKind> credential_kind;
 std::optional<std::string> credential_kind_value;
 std::string tenant_id;
 std::string client_id;
 std::string client_secret;
 ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
 for (const auto& kv : options_items) {
 if (kv.first == "blob_storage_authority") {
 blob_storage_authority = kv.second;
 } else if (kv.first == "dfs_storage_authority") {
 dfs_storage_authority = kv.second;
 } else if (kv.first == "credential_kind") {
@felipecrvfelipecrvMar 4, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.
@koukouMar 5, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?
@felipecrvfelipecrvMar 5, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.
@koukouMar 6, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.
@felipecrvfelipecrvMar 6, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:
  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)
How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.
@felipecrvfelipecrvMar 7, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
Sure. That looks good.
 if (kv.second == "default") {
 credential_kind = CredentialKind::kDefault;
 } else if (kv.second == "anonymous") {
 credential_kind = CredentialKind::kAnonymous;
 } else if (kv.second == "workload_identity") {
 credential_kind = CredentialKind::kWorkloadIdentity;
 } else {
 // Other credential kinds should be inferred from the given
 // parameters automatically.
 return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
 }
 credential_kind_value = kv.second;
 } else if (kv.first == "tenant_id") {
 tenant_id = kv.second;
 } else if (kv.first == "client_id") {
 client_id = kv.second;
 } else if (kv.first == "client_secret") {
 client_secret = kv.second;
 } else if (kv.first == "enable_tls") {
 ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
 if (enable_tls) {
 blob_storage_scheme = "https";
 dfs_storage_scheme = "https";
 } else {
 blob_storage_scheme = "http";
 dfs_storage_scheme = "http";
 }
 } else {
 return Status::Invalid(
 "Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
 }
 }

 if (credential_kind) {
 if (!account_key.empty()) {
 return Status::Invalid("Password must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with credential_kind=",
 *credential_kind_value);
 }

 switch (*credential_kind) {
 case CredentialKind::kAnonymous:
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 break;
 case CredentialKind::kWorkloadIdentity:
 RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
 break;
 default:
 // Default credential
 break;
 }
 } else {
 if (!account_key.empty()) {
 // With password
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with password");
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with password");
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with password");
 }
 RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
 } else {
 // Without password
 if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
 // No related parameters
 if (account_name.empty()) {
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 } else {
 // Default credential
 }
 } else {
 // One or more tenant_id, client_id or client_secret are specified
 if (client_id.empty()) {
 return Status::Invalid("client_id must be specified");
 }
 if (tenant_id.empty() && client_secret.empty()) {
 RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
 } else if (!tenant_id.empty() && !client_secret.empty()) {
 RETURN_NOT_OK(
 ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
 } else {
 return Status::Invalid("Both of tenant_id and client_secret must be specified");
 }
 }
 }
 }
 return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
 std::string* out_path) {
 AzureOptions options;
 options.ExtractFromUriSchemeAndHierPart(uri, out_path);
 RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
 return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
 std::string* out_path) {
 arrow::internal::Uri uri;
 RETURN_NOT_OK(uri.Parse(uri_string));
 return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
 // TODO(GH-38598): update here when more auth methods are added.
 const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down

56 changes: 56 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
 friend class TestAzureOptions;

 /// \brief The name of the Azure Storage Account being accessed.
 ///
 /// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
 AzureOptions();
 ~AzureOptions();

 private:
 void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path);
 Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

 public:
 /// \brief Construct a new AzureOptions from an URI.
 ///
 /// Supported formats:
 ///
 /// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
 /// [/\<container\>[/\<path\>]]
 /// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
 /// [/path]
 /// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
 /// [/\<container\>[/path]]
 /// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
 ///
 /// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
 /// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
 ///
 /// 3. is for Azure Blob Storage compatible service including Azurite.
 ///
 /// 4. is a shorter version of 1. and 2.
 ///
 /// Note that there is no difference between abfs and abfss. HTTPS is
 /// used with abfs by default. You can force to use HTTP by specifying
 /// "enable_tls=false" query.
 ///
 /// Supported query parameters:
 ///
 /// * blob_storage_authority: Set AzureOptions::blob_storage_authority
 /// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
 /// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
 /// * credential_kind: One of "default", "anonymous",
 /// "workload_identity". If "default" is specified, it's just
 /// ignored. If "anonymous" is specified,
 /// AzureOptions::ConfigureAnonymousCredential() is called. If
 /// "workload_identity" is specified,
 /// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
 /// * tenant_id: You must specify "client_id" and "client_secret"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_id: If you don't specify "tenant_id" and
 /// "client_secret",
 /// AzureOptions::ConfigureManagedIdentityCredential() is
 /// called. If you specify "tenant_id" and "client_secret" too,
 /// AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_secret: You must specify "tenant_id" and "client_id"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
 std::string* out_path);
 static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

 Status ConfigureDefaultCredential();
 Status ConfigureAnonymousCredential();
 Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()#40325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

kou
merged 9 commits into

from

Mar 11, 2024
 Merged


GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()
#40325
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
704b69e

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemF…


kou Mar 3, 2024
9e7d75e

Update example URIs


kou Mar 3, 2024
81076bc

Add the document URI for abfs:// and abfss://


kou Mar 3, 2024
fb692de

Remove needless empty container check


kou Mar 5, 2024
376017f

Remove needless options_map


kou Mar 5, 2024
b8e6b83

Implement discussed spec


kou Mar 7, 2024
5927be8

Escape special characters


kou Mar 7, 2024
0a76645

Fix lint error


kou Mar 7, 2024
58d948f

Fix lint error


kou Mar 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


166 changes: 166 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -65,6 +65,172 @@ AzureOptions::AzureOptions() = default;

AzureOptions::~AzureOptions() = default;

void AzureOptions::ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path) {
 const auto host = uri.host();
 std::string path;
 if (arrow::internal::EndsWith(host, blob_storage_authority)) {
 account_name = host.substr(0, host.size() - blob_storage_authority.size());
 path = internal::RemoveLeadingSlash(uri.path());
 } else if (arrow::internal::EndsWith(host, dfs_storage_authority)) {
 account_name = host.substr(0, host.size() - dfs_storage_authority.size());
 path = internal::ConcatAbstractPath(uri.username(), uri.path());
 } else {
 account_name = uri.username();
 const auto port_text = uri.port_text();
 if (host.find(".") == std::string::npos && port_text.empty()) {
 // abfs://container/dir/file
 path = internal::ConcatAbstractPath(host, uri.path());
 } else {
 // abfs://host.domain/container/dir/file
 // abfs://host.domain:port/container/dir/file
 // abfs://host:port/container/dir/file
 std::string host_port = host;
 if (!port_text.empty()) {
 host_port += ":" + port_text;
 }
 blob_storage_authority = host_port;
 dfs_storage_authority = host_port;
 path = internal::RemoveLeadingSlash(uri.path());
 }
 }
 if (out_path != nullptr) {
 *out_path = path;
 }
}

Status AzureOptions::ExtractFromUriQuery(const arrow::internal::Uri& uri) {
 const auto account_key = uri.password();
 std::optional<CredentialKind> credential_kind;
 std::optional<std::string> credential_kind_value;
 std::string tenant_id;
 std::string client_id;
 std::string client_secret;
 ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
 for (const auto& kv : options_items) {
 if (kv.first == "blob_storage_authority") {
 blob_storage_authority = kv.second;
 } else if (kv.first == "dfs_storage_authority") {
 dfs_storage_authority = kv.second;
 } else if (kv.first == "credential_kind") {
@felipecrvfelipecrvMar 4, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.
@koukouMar 5, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?
@felipecrvfelipecrvMar 5, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.
@koukouMar 6, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.
@felipecrvfelipecrvMar 6, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:
  • nothing (use default auth chain)
  • ?anonymous
  • ?use_workload_identity
  • ?account_key=<ACCOUNT_KEY>
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)
How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
  • nothing (use default auth chain) -> nothing or ?credential_kind=default
  • ?anonymous -> ?credential_kind=anonymous
  • ?use_workload_identity -> ?credential_kind=workload_identity
  • ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
  • ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
  • ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)
@koukouMar 7, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.
@felipecrvfelipecrvMar 7, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?
Sure. That looks good.
 if (kv.second == "default") {
 credential_kind = CredentialKind::kDefault;
 } else if (kv.second == "anonymous") {
 credential_kind = CredentialKind::kAnonymous;
 } else if (kv.second == "workload_identity") {
 credential_kind = CredentialKind::kWorkloadIdentity;
 } else {
 // Other credential kinds should be inferred from the given
 // parameters automatically.
 return Status::Invalid("Unexpected credential_kind: '", kv.second, "'");
 }
 credential_kind_value = kv.second;
 } else if (kv.first == "tenant_id") {
 tenant_id = kv.second;
 } else if (kv.first == "client_id") {
 client_id = kv.second;
 } else if (kv.first == "client_secret") {
 client_secret = kv.second;
 } else if (kv.first == "enable_tls") {
 ARROW_ASSIGN_OR_RAISE(auto enable_tls, ::arrow::internal::ParseBoolean(kv.second));
 if (enable_tls) {
 blob_storage_scheme = "https";
 dfs_storage_scheme = "https";
 } else {
 blob_storage_scheme = "http";
 dfs_storage_scheme = "http";
 }
 } else {
 return Status::Invalid(
 "Unexpected query parameter in Azure Blob File System URI: '", kv.first, "'");
 }
 }

 if (credential_kind) {
 if (!account_key.empty()) {
 return Status::Invalid("Password must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with credential_kind=",
 *credential_kind_value);
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with credential_kind=",
 *credential_kind_value);
 }

 switch (*credential_kind) {
 case CredentialKind::kAnonymous:
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 break;
 case CredentialKind::kWorkloadIdentity:
 RETURN_NOT_OK(ConfigureWorkloadIdentityCredential());
 break;
 default:
 // Default credential
 break;
 }
 } else {
 if (!account_key.empty()) {
 // With password
 if (!tenant_id.empty()) {
 return Status::Invalid("tenant_id must not be specified with password");
 }
 if (!client_id.empty()) {
 return Status::Invalid("client_id must not be specified with password");
 }
 if (!client_secret.empty()) {
 return Status::Invalid("client_secret must not be specified with password");
 }
 RETURN_NOT_OK(ConfigureAccountKeyCredential(account_key));
 } else {
 // Without password
 if (tenant_id.empty() && client_id.empty() && client_secret.empty()) {
 // No related parameters
 if (account_name.empty()) {
 RETURN_NOT_OK(ConfigureAnonymousCredential());
 } else {
 // Default credential
 }
 } else {
 // One or more tenant_id, client_id or client_secret are specified
 if (client_id.empty()) {
 return Status::Invalid("client_id must be specified");
 }
 if (tenant_id.empty() && client_secret.empty()) {
 RETURN_NOT_OK(ConfigureManagedIdentityCredential(client_id));
 } else if (!tenant_id.empty() && !client_secret.empty()) {
 RETURN_NOT_OK(
 ConfigureClientSecretCredential(tenant_id, client_id, client_secret));
 } else {
 return Status::Invalid("Both of tenant_id and client_secret must be specified");
 }
 }
 }
 }
 return Status::OK();
}

Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
 std::string* out_path) {
 AzureOptions options;
 options.ExtractFromUriSchemeAndHierPart(uri, out_path);
 RETURN_NOT_OK(options.ExtractFromUriQuery(uri));
 return options;
}

Result<AzureOptions> AzureOptions::FromUri(const std::string& uri_string,
 std::string* out_path) {
 arrow::internal::Uri uri;
 RETURN_NOT_OK(uri.Parse(uri_string));
 return FromUri(uri, out_path);
}

bool AzureOptions::Equals(const AzureOptions& other) const {
 // TODO(GH-38598): update here when more auth methods are added.
 const bool equals = blob_storage_authority == other.blob_storage_authority &&
Expand Down

56 changes: 56 additions & 0 deletions
cpp/src/arrow/filesystem/azurefs.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -45,6 +45,7 @@ class DataLakeServiceClient;
namespace arrow::fs {

class TestAzureFileSystem;
class TestAzureOptions;

/// Options for the AzureFileSystem implementation.
///
Expand All@@ -59,6 +60,8 @@ class TestAzureFileSystem;
///
/// Functions are provided for explicit configuration of credentials if that is preferred.
struct ARROW_EXPORT AzureOptions {
 friend class TestAzureOptions;

 /// \brief The name of the Azure Storage Account being accessed.
 ///
 /// All service URLs will be constructed using this storage account name.
Expand DownExpand Up@@ -123,6 +126,59 @@ struct ARROW_EXPORT AzureOptions {
 AzureOptions();
 ~AzureOptions();

 private:
 void ExtractFromUriSchemeAndHierPart(const arrow::internal::Uri& uri,
 std::string* out_path);
 Status ExtractFromUriQuery(const arrow::internal::Uri& uri);

 public:
 /// \brief Construct a new AzureOptions from an URI.
 ///
 /// Supported formats:
 ///
 /// 1. abfs[s]://[:\<password\>@]\<account\>.blob.core.windows.net
 /// [/\<container\>[/\<path\>]]
 /// 2. abfs[s]://\<container\>[:\<password\>]@\<account\>.dfs.core.windows.net
 /// [/path]
 /// 3. abfs[s]://[\<account[:\<password\>]@]\<host[.domain]\>[\<:port\>]
 /// [/\<container\>[/path]]
 /// 4. abfs[s]://[\<account[:\<password\>]@]\<container\>[/path]
 ///
 /// 1. and 2. are compatible with the Azure Data Lake Storage Gen2 URIs:
 /// https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri
 ///
 /// 3. is for Azure Blob Storage compatible service including Azurite.
 ///
 /// 4. is a shorter version of 1. and 2.
 ///
 /// Note that there is no difference between abfs and abfss. HTTPS is
 /// used with abfs by default. You can force to use HTTP by specifying
 /// "enable_tls=false" query.
 ///
 /// Supported query parameters:
 ///
 /// * blob_storage_authority: Set AzureOptions::blob_storage_authority
 /// * dfs_storage_authority: Set AzureOptions::dfs_storage_authority
 /// * enable_tls: If it's "false" or "0", HTTP not HTTPS is used.
 /// * credential_kind: One of "default", "anonymous",
 /// "workload_identity". If "default" is specified, it's just
 /// ignored. If "anonymous" is specified,
 /// AzureOptions::ConfigureAnonymousCredential() is called. If
 /// "workload_identity" is specified,
 /// AzureOptions::ConfigureWorkloadIdentityCredential() is called.
 /// * tenant_id: You must specify "client_id" and "client_secret"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_id: If you don't specify "tenant_id" and
 /// "client_secret",
 /// AzureOptions::ConfigureManagedIdentityCredential() is
 /// called. If you specify "tenant_id" and "client_secret" too,
 /// AzureOptions::ConfigureClientSecretCredential() is called.
 /// * client_secret: You must specify "tenant_id" and "client_id"
 /// too. AzureOptions::ConfigureClientSecretCredential() is called.
 static Result<AzureOptions> FromUri(const arrow::internal::Uri& uri,
 std::string* out_path);
 static Result<AzureOptions> FromUri(const std::string& uri, std::string* out_path);

 Status ConfigureDefaultCredential();
 Status ConfigureAnonymousCredential();
 Status ConfigureAccountKeyCredential(const std::string& account_key);
Expand Down
Loading