[R] Documentation for json_credentials is misleading

[asfimport]

For authenticating with GCS via a JSON credentials file, the documentation under ?FileSystem for GcsFileSystem$create() says:

 

• {}json_credentials{}: optional string for authentication. Point to a JSON credentials file downloaded from GCS.

   

  Additionally, the GCS Authentication section of Working with Cloud Storage (S3, GCS) in the file system vignette says:

  or {}json_credentials{}, to reference a downloaded credentials file.
   

  Both of these seem to imply that json_credentials expects a path to a JSON credentials file downloaded from GCP. However, when a file path is provided you get an invalid argument error:

   

  > bucket <- gs_bucket(bucket = 'pinned_data',json_credentials = json_path)
  > bucket$ls(recursive = TRUE)
  Error: Invalid: google::cloud::Status(INVALID_ARGUMENT: Permanent error in ListObjects: Invalid ServiceAccountCredentials,parsing failed on data loaded from memory). Detail: [errno 22] Invalid argument

   

  However, if you pass a string containing the raw JSON from the file itself, the above code snippet works and returns the names of the objects in the bucket.

  Both sections of the documentation should be clarified to explicitly say that the argument expects the actual JSON rather than a file path to the JSON file.

Reporter: Joran Elias

_{Note: This issue was originally created as ARROW-17894. Please see the migration documentation for further details.}

[asfimport]

Carl Boettiger / @cboettig:
Unfortunately I still get an error when passing the raw JSON

gs$ls()
Error: Invalid: google::cloud::Status(INVALID_ARGUMENT: Permanent error in ListObjects: Invalid ServiceAccountCredentials: could not parse PEM to get private key ). Detail: [errno 22] Invalid argument

[asfimport]

Joran Elias:
@cboettig Sorry to see this has bitten you too! Not sure what the difference would be between our setups, but in case it helps, what worked for me was reading in the GCP json file via readr::read_file so that it's a single character object, and then passing that object to the json_credentials argument. Also, I'm using R on a Ubuntu VM via RStudio Workbench.

[asfimport]

Nicola Crane / @thisisnic:
Thanks for reporting this @cboettig and for adding a potential solution there [~joran_elias]! If this works for you @cboettig, do you fancy submitting a PR to update those docs?

[asfimport]

Carl Boettiger / @cboettig:
I no longer get the error when I use read_file. Unfortunately I don't seem to get access to the bucket either:

bucket <- arrow::gs_bucket("neon-is-transition-output",
 json_credentials = readr::read_file("service_account.json"))
bucket$ls()

is just empty, no error, no files. Not sure what is going wrong or how to debug...

Note that I can use this same service_account.json to list files in this same bucket using rclone just fine, and I can use the HMAC credentials to get arrow::s3_bucket() to successfully list the bucket contents too, so I don't think the problem is with my bucket or my service_account.json. 

Note that arrow::s3_bucket() strategy with HMAC credentials is not a viable work-around for google cloud since Google's implementation of S3 suffers from numerous documented deviations from the S3 standard that cause most operations to fail, eg see https://gist.github.com/harshavardhana/5d4d7410afb02ec9837e2b4d82a932e7.

So I'd really love native GCS support in arrow. (The bucket indicated above contains data from the National Science Foundation's National Ecological Observatory Network, nearly half-billion taxpayer dollars going to create this data and they chose to host it on Google storage... so we'd love to be able to access it with an amazing tool like arrow ...)

[asfimport]

Bryce Mecum / @amoeba:
Hi Carl, I just ran through similar steps as your above with my own bucket and credentials and it worked as expected.

 

library(arrow)

arrow_info()$version
# [1] ‘10.0.1’

bucket <- gs_bucket("my-gcs-bucket",
 json_credentials = readr::read_file("my-service-account.json"))
bucket$ls()
# [1] "starwars.csv" "starwars.parquet" 

I wonder if some detail of how we generated our service accounts is different. I also set up Application Default Credentials instead of a Service Account (like Google themselves and our docs suggest), and read_parquet and open_dataset work against GCS URIs just fine.

 

I do want to see you get this working so we can either chat 1:1 or you could file a [new issue on GitHub|github.com/apache/arrow/issues/new] and we could go from there.

[amoeba]

take

[amoeba]

Hey @cboettig, I've written up a standalone issue for what you're seeing at #34525 and think we can follow up there.

Once #34524 merges this issue can be closed.