GH-40037: [C++][FS][Azure] Make attempted reads and writes against directories fail fast

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)

[Tom-Newton]

Rationale for this change

Prevent confusion if a user attempts to read or write a directory.

What changes are included in this PR?

• Make ObjectAppendStream::Flush a noop if ObjectAppendStream::Init has not run successfully. This avoids an unhandled error when the destructor calls flush.
• Check blob properties for directory marker metadata when initialising ObjectInputFile or ObjectAppendStream.
• When initialising ObjectAppendStream call GetFileInfo if it is a flat namespace account.

Are these changes tested?

Add new tests DisallowReadingOrWritingDirectoryMarkers and DisallowCreatingFileAndDirectoryWithTheSameName to cover the new fail fast behaviour.
Also updated WriteMetadata to ensure that my changes to Flush didn't break setting metadata without calling Write on the stream.

Are there any user-facing changes?

Yes. Invalid read and write operations will now fail fast and gracefully. Previously could get into a confusing state where there were files and directories at the same path and there were some un-graceful failures.

• Closes: [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037

[Tom-Newton]

@felipecrv I expect you will be interested in reviewing this.

[felipecrv]

You can inject AzureFileSystem *azure_file_system here and not have to allocate a closure for this. You would call AzureFileSystem::Impl::EnsureNotFlatNamespaceDirectory(location) via azure_file_system->impl_ (accessible because the handles produced by the azure file system can be friends with the filesystem class).

[Tom-Newton]

Thanks for the extra info. I was planning to do this but I was struggling with the friends thing.

[felipecrv]

The trick is to forward-declare the class as well.

diff --git a/cpp/src/arrow/filesystem/azurefs.h b/cpp/src/arrow/filesystem/azurefs.h
index 2a131e40c..d48ef9dd7 100644
--- a/cpp/src/arrow/filesystem/azurefs.h
+++ b/cpp/src/arrow/filesystem/azurefs.h
@@ -44,6 +44,7 @@ classDataLakeServiceClient;
namespacearrow::fs {
+classObjectAppendStream;
classTestAzureFileSystem;
/// Options for the AzureFileSystem implementation.
@@ -180,6 +181,7 @@ classARROW_EXPORT AzureFileSystem : public FileSystem {
explicitAzureFileSystem(std::unique_ptr<Impl>&& impl);
+ friendclassObjectAppendStream;
friendclassTestAzureFileSystem;
voidForceCachedHierarchicalNamespaceSupport(int hns_support);

[Tom-Newton]

I think my main problem was that ObjectAppendStream is defined inside an anonymous namespace but I still haven't got it working as you describe.

Are you suggesting to use AzureFileSystem *azure_file_system or AzureFileSystem:Impl *azure_file_system as the argument to ObjectAppendStream::Impl. I don't know how I can get a AzureFileSystem pointer from inside AzureFileSystem::Impl and using AzureFileSystem::Impl as the argument leads to incomplete type errors which I don't think I can avoid.

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

Sorry about my lacking C++ knowledge here.

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[Tom-Newton]

Thanks for explaining

[felipecrv]

Also if you wouldn't mind I would be interested to know what the disadvantage of a lambda function is compared to what you proposed.

To create the std::function, you heap allocate an object with copies of the values in the capture list and generate a lot more extra code in the binary:

class function {
T valuesfromthecpapturelist;
RetType operator()(ArgsType ...) {...};
}

When you think about an std::function this way (a pair of context data and a function), you realize the class you already serves that purpose.

But hey, this is becoming challenging, so I won't hold the PR anymore because of this. Moving to Init() was a big step in the right direction.

[felipecrv]

This is causing the linter to fail @Tom-Newton. Please fix and I will merge.

[Tom-Newton]

Fixed

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

My apologies for the noise. Looking into this now.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 8a62f30.

There was 1 benchmark result with an error:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-02-21 17:26:57Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
  • dataframe-to-table (R) with dataset=type_integers, language=R

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[austin3dickey]

That should be the last one. Sorry again about the noise!

[felipecrv]

@austin3dickey no worries :)