GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri()

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

• GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

⚠️ GitHub issue #40028has been automatically assigned in GitHub to PR creator.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

Thanks for the note. I've added the documentation URL as a comment.

A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

• nothing (use default auth chain)
• ?anonymous
• ?use_workload_identity
• ?account_key=<ACCOUNT_KEY>
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

• nothing (use default auth chain) -> nothing or ?credential_kind=default
• ?anonymous -> ?credential_kind=anonymous
• ?use_workload_identity -> ?credential_kind=workload_identity
• ?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)
• ?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)
• ?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

@kou@Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_urlhttps://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

• abfs[s]://<container>/<path> (according to fsspec)
• abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
• abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
• az://<container>/<path>
• adl://<container>/<path>
• azure://<container>/<path>
• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

OK. I'll implement the discussed spec.

Implemented:

Supported formats:

1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
4. abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

• enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
  parameters.

Removed query parameters:

• blob_storage_scheme: Replaced with enable_tls.
• dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

• credential_kind: Accepts only default, anonymous and
  workload_identity.

I'll merge this in the next week if nobody objects it.

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.