[C++] Cannot write dataset to S3FileSystem if bucket already exists

[asfimport]

I'm trying to write a parquet file to an existing S3 bucket using the new S3FileSystem interface. However, this is failing with an AWS Access Denied error (I do have necessary access). It appears to be trying to recreate the bucket which already exists.

import numpy as np
import pyarrow as pa
from pyarrow import fs
import pyarrow.dataset as ds

s3 = fs.S3FileSystem(region="us-west-2")
table = pa.table({"a": range(10), "b": np.random.randn(10), "c": [1, 2] * 5})
ds.write_dataset(
 table,
 "my-bucket/test.parquet",
 format="parquet",
 filesystem=s3,
)

OSError: When creating bucket 'my-bucket': AWS Error [code 15]: Access Denied

I'm seeing the same behavior using S3FileSystem.create_dir when recursive=True.

s3.create_dir("my-bucket/test_dir/", recursive=True) # Fails
s3.create_dir("my-bucket/test_dir/", recursive=False) # Succeeds

 

Reporter: Caleb Overman / @caleboverman
Assignee: Weston Pace / @westonpace

PRs and other links:

• GitHub Pull Request #11136

_{Note: This issue was originally created as ARROW-13685. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
Arrow will always try and create the bucket (unless creating a directory with recursive=False). It then checks to see if the error is BUCKET_ALREADY_EXISTS or BUCKET_ALREADY_OWNED_BY_YOU and, if it is, it treats it as a success.

Do you perhaps have some more sophisticated permissions set on the bucket? Maybe your user is allowed to create files inside the bucket but not allowed to modify the bucket and so S3 is returning a permission denied error instead of BUCKET_ALREADY_EXISTS?

[asfimport]

Caleb Overman / @caleboverman:
You are correct. I've got the following permissions on the bucket:

"s3:PutObject",
"s3:GetObjectAcl",
"s3:GetObject",
"s3:ListBucket",
"s3:PutObjectTagging",
"s3:DeleteObject",
"s3:GetObjectVersion"

[asfimport]

Weston Pace / @westonpace:
Thanks for the detailed information. I confirm I get the same error when I setup with the permissions you've specified. Then when I run as a root user everything works correctly. It appears MinIO accepts the same kinds of policy files as S3 so I should be able to create a unit/regression test for this scenario.

[asfimport]

Caleb Overman / @caleboverman:
Thank you for looking into this. I'm not familiar with the filesystem implementation, but if there's anything I can do to help please let me know!

[asfimport]

Kyle Hamlin:
I'm also now running into this issue after having the following issue resolved in 5.0.0: https://issues.apache.org/jira/browse/ARROW-13228

@westonpace Curious what the intended resolution is as it's pretty uncommon to give create bucket permission to a data, ETL, or ML application. I think it would be preferable to have bucket creation separate from writing to a bucket/prefix or to have something like create_bucket=True as a parameter to write_dataset.

[asfimport]

Weston Pace / @westonpace:
Hmm, datasets are filesystem-agnostic so I'd prefer to avoid create_bucket as a dataset option. It could be an option added to the S3 filesystem (create_bucket_if_not_exists or something) but I think my preferred fix for this issue would be to modify S3FileSystem::CreateBucket so that it first does a check to see if the bucket exists. I suspect the current implementation is the way that it is just to reduce complexity (why make more calls than needed) but it isn't working here so the more complex solution is warranted. I'll go ahead and put a PR up for this soon. The trickiest part will probably be the regression test more than the implementation.

[asfimport]

Kyle Hamlin:
@westonpace Checking for a buckets existence sound like a fine solution to me. Do you know what extra, if any, permissions would be required for that check?

[asfimport]

Weston Pace / @westonpace:
It appears HeadBucket is the operation for this and it only requires permission to perform the "s3:ListBucket" action so it should suffice for @caleboverman's case.

[asfimport]

Weston Pace / @westonpace:
Issue resolved by pull request 11136
#11136

[asfimport]

Mark van der Broek:
Not checking for existing buckets actually results in another issue if multiple processes try to create this bucket at exactly the same time.

 

I get the following error:

OSError: When creating bucket '': AWS Error [code 100]: Unable to parse ExceptionName: OperationAborted Message: A conflicting conditional operation is currently in progress against this resource. Please try again.

[asfimport]

Weston Pace / @westonpace:
[~1mvdb] I expect you would've been able to get this error with the old path too. I didn't remove the old behavior or any checks, I just added an additional check. It might be best to create a new JIRA. It's not clear to me that translating error code 100 into "bucket already exists" is the correct thing to do. We could maybe create an auto-retry if we see "Please try again" but from some googling it appears this condition could take 15 minutes or more to resolve in some situations (e.g. when creating a bucket with the same name as a recently deleted bucket).