GH-38597: [C++] Implement GetFileInfo(selector) for Azure filesystem

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)

[felipecrv]

Rationale for this change

Part of Azure FS implementation.

What changes are included in this PR?

The version of GetFileInfo that takes a prefix and can optionally recurse into directories.

Are these changes tested?

By unit tests present in this PR. Separate from this PR, I'm thinking of way to fuzz-test the FS API.

• Closes: [C++] Implement GetFileInfo selector for Azure filesystem #38597

[felipecrv]

@kou@Tom-Newton

[felipecrv]

TODO:

• Leverage hierarchical support if enabled for the account
• Introduce an explicit std::stack to remove stack overflow risk when listing very deep filesystems

[kou]

(I'll review GetFileInfoWithSelector() and GetFileInfoWithSelectorFromContainer() logic carefully later.)

[kou]

How about moving this to path_util.{cc,h}?
Because extracting basename from a path is a generic path operation.
We will be able to use GetAbstractPathParent() to implement this like return GetAbstractPathParent(s).second;.

[pitrou]

+1 for factoring out generic path operations, and also unit-test them.

[felipecrv]

The problem with path_util.h functions is that they don't follow any concept of path normalization for inputs and have to check the input for many variations, this BasenameView works well in this context, but not in a general context.

[felipecrv]

I'm changing it to this:

static std::string_view BasenameView(std::string_view s) {
DCHECK(!internal::HasTrailingSlash(s));
auto offset = s.find_last_of(internal::kSep);
auto result = (offset == std::string_view::npos) ? s : s.substr(offset);
DCHECK(!result.empty() && result.back() != internal::kSep);
return result;
}

[kou]

I don't object them but why do you want to specify them explicitly?
Our other methods don't specify them explicitly. (They use the default value.)

[felipecrv]

I want all the wiring to be in place if we need to specify the page_size_hint which I'm pretty sure we will need to tweak when this is used in practice.

[kou]

Should we pass page_size_hint here too?

[felipecrv]

page_size_hint only applies to the pagination of blobs. If I were to pass page size here, it would be a separate constant, but I thought this would be an overkill.

[kou]

Is this really happen?
emptydir/ is a case of this in the test, right?
In this case, emptydir/ may exist in Blobs (with options.Prefix = emptydir/) or BlobPrefixes (with no options.Prefix) but doesn't exit in both of them. (I checked this with gdb.)

[felipecrv]

This is the direction the whole implementation will move towards. CreateDir should create a marker blob, so we can check that an empty directory exists after it's created or after its contents are deleted.

As @pitrou said in another comment, all the Filesystem implementations should behave the same way.

[kou]

Can we use ListBlobs() instead when select.max_recursion == INT32_MAX as an optimization?

[felipecrv]

That would require that we build a trie of all the blob names to list them hierarchically. I'm open to doing it, but I think we can delay doing it until there is clear indication that it would help a concrete use-case.

[Tom-Newton]

I reviewed the tests, so I'm happy with the behaviour. I haven't had a chance to understand and review the implementation yet.

[pitrou]

General question: was it considered to have a namespace alias at the top of this file such as namespace BlobsModels = Azure::Storage::Blobs::Models;?

[felipecrv]

Something I prefer to do once all PRs land because it would be a noisy change.

[felipecrv]

I rebased to fix a conflict, but the new changes are on the last two commits.

[kou]

+1

Could you fix a lint failure?

[felipecrv]

I will rebase and force-push to see if all CI passes now.

[felipecrv]

CI failure is unrelated so I'm merging this.

[kou]

@felipecrv Did you use our merge script?
https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 476c78f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 32 possible false positives for unstable benchmarks that are known to sometimes produce them.

[felipecrv]

@felipecrv Did you use our merge script? https://github.com/apache/arrow/tree/main/dev#how-to-merge-a-pull-request

GH-38597's milestone isn't set. In general, it's set by the merge script.

Oh no. Sorry. I had no idea. I will use it from now on. Anything we can do to make this right for this PR?

[kou]

I set GH-37597's milestone manually. :-)