GH-38772: [C++] Implement directory semantics even when the storage account doesn't support HNS

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

[felipecrv]

Rationale for this change

The FileSystem implementation based on Azure Blob Storage should implement directory operations according to filesystem semantics. When Hierarchical Namespace (HNS) is enabled, we can rely on Azure Data Lake Storage Gen 2 APIs implementing the filesystem semantics for us, but when all we have is the Blobs API, we should emulate it.

What changes are included in this PR?

• Skip fewer tests
• Re-implement GetFileInfo using ListBlobsByHierarchy instead of ListBlobs
• Re-implement CreateDir with an upfront HNS support check instead of falling back to Blobs API after an error
• Add comprehensive tests to CreateDir
• Add HasSubmitBatchBug to check if a test inside any scenario is affected by a certain Azurite issue
• Implement DeleteDir to work properly on flat namespace storage accounts (non-HNS accounts)

Are these changes tested?

Yes. By existing and new tests added by this PR itself.

• Closes: [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772

[felipecrv]

@Tom-Newton

[Tom-Newton]

I have not reviewed everything yet. I have a few comments, but generally I think it looks great so far.

[Tom-Newton]

I think it would be a good idea to put some metadata on all directory marker blobs we create. That would allow us to add a simple checks in ObjectInputFile::Init and ObjectAppendStream::Init to ensure they are not being initialised as directories.

Additionally I think this will not be compatible with adlfs if it does not add metadata to directory markers. I think its very likely that people will end up using a combination of this arrow filesystem and adlfs.

[felipecrv]

Sure. I would like to be as compatible as possible with existing specs. But to clarify, what do you mean when you say adlfs? Don't you mean abfs instead [1]?

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

I think there is a confusion between abfs (filesystem semantics over the Blobs API) and adlfs (the separate service that Azure created that can also be accessed via the Blobs API with some restrictions).

The fsspec repo seems to be linking both abfs and adlfs to the same thing, but they shouldn't be considered the same thing: https://github.com/fsspec/filesystem_spec/blob/0ffe06cb767456b7c13904b57ec1c3ca60d53eae/docs/source/api.rst?plain=1#L229

Tell me what I'm missing here because my experience with Azure is restricted just to reading the docs (a lot of them) recently.

[Tom-Newton]

Sorry, in this case I mean the adlfs pip repo. It's the Azure fsspec implementation.

[Tom-Newton]

More specifically I think we should add some metadata which will be compatible such that https://github.com/fsspec/adlfs/blob/32132c4094350fca2680155a5c236f2e9f991ba5/adlfs/spec.py#L855-L870 can correctly detect the directories.

[felipecrv]

I forgot this, but I will add this change now.

[felipecrv]

@Tom-Newton@kou I pushed changes based on your feedback. I would love to merge it before tomorrow's release cut.

[kou]

Could you check CI failures?

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3814

[ RUN ] TestAzuriteFileSystem.DeleteDirContentsSuccessDirectory
/arrow/cpp/src/arrow/filesystem/test_util.cc:143: Failure
Expected equality of these values:
info.type()
Which is: FileType::Directory
type
Which is: FileType::NotFound
For path 'hfqilxw9uhk7xqbathiwbudo4ci9dp9w/0iav1y3m4u53d5wvtp8g2j5af1o4s694'

https://github.com/apache/arrow/actions/runs/7414494286/job/20175680211?pr=39361#step:6:3680

[ RUN ] TestAzuriteFileSystem.DeleteDirSuccessNonexistent
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc:1301: Failure
Failed
'fs()->DeleteDir(directory_path)' failed with IOError: Path does not exist 'go9qhglaypjhxnprrxc79a0e3xtceml1/bwumjdfql0n8hizvv1kd3jn75ucrlw4u'. Detail: [errno 2] No such file or directory

[felipecrv]

@kou done. These turned out to be Azurite-only tests that needed updating since the semantics are now different than what was assumed when the tests were written.

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit aae6fa4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.