Describe the enhancement requested
GcsOptions::FromServiceAccountCredentials (cpp/src/arrow/filesystem/gcsfs.h:146) is the
one credential factory on GcsOptions with no Python binding, so a caller holding
aip/4112 service-account JSON has no way to construct a GcsFileSystem from it.
python/pyarrow/includes/libarrow_fs.pxd declares the other four (Defaults, Anonymous,
FromAccessToken, FromImpersonatedServiceAccount, lines 229-240), and the
credential-routing chain inside GcsFileSystem.__init__ (python/pyarrow/_gcsfs.pyx:103-131,
method at line 88) selects among them. No constructor keyword reaches the fifth. Line
references are against d048f71; behaviour checked on pyarrow 25.0.1.
The R bindings already have it. GcsFileSystem$create() takes json_credentials, either the
JSON itself or a path to a file holding it (r/R/filesystem.R:611-674), and routes it to
GcsOptions::FromServiceAccountCredentials (r/src/filesystem.cpp:415-417); it also rejects
json_credentials alongside access_token, which is the mutual exclusion proposed below. That
landed in #34524 (GH-34421) in March 2023. So the two bindings have complementary holes: R has
service-account JSON and lacks impersonation (#32207), Python has impersonation
(target_service_account) and lacks service-account JSON.
Two routes are open today to an application holding that JSON in memory, and each gives
something up:
- Write it to a file and point
GOOGLE_APPLICATION_CREDENTIALS at it. That puts the private
key on disk, and it configures the process rather than the filesystem object, so two
filesystems in one process cannot use different service accounts. The docstring on this
function names the case it exists for: "Using this function may be useful when the json
object is obtained from a Cloud Secret Manager or a similar service." - Mint an access token out of band and pass
access_token + credential_token_expiration.
Per FromAccessToken's docstring those tokens are time limited and the caller refreshes
them manually, and the Python API offers no way to replace the token on an existing
GcsFileSystem, so a refresh means constructing a new one.
Concrete case: omniload (panodata/omniload#233) accepts GCS service-account JSON as either a
path or a base64 blob from a secret store. Its S3 and R2 transports moved to pyarrow.fs for
the per-file open latency (on that sibling transport, 8.68 ms/file through s3fs against
3.49 ms/file through Arrow, on the reader path that opens one file object per file). GCS
stayed on gcsfs, because the swap would drop that credential mode.
Proposed shape, following the existing chain:
- Declare
FromServiceAccountCredentials in libarrow_fs.pxd beside the other four. - Add a
GcsFileSystem.__init__ keyword taking the aip/4112 JSON as a string, in the same
base-credential branch as anonymous and access_token and mutually exclusive with both.
service_account_json reads more clearly than service_account_credentials, since the
value is serialized JSON rather than a credentials object; the C++ accessor calls it
json_credentials. It would stay composable with target_service_account by passing the
resulting credentials to FromImpersonatedServiceAccount, as the constructor already does
for the other base credentials. - Pickling would need a policy decision, and I have not assumed one.
__reduce__ round-trips
access_token today (_gcsfs.pyx:183), but a long-lived private key is a different
proposition and declining to serialize this mode is a reasonable answer. If it should
round-trip through the existing __reduce__ design, the pxd would also need
GcsCredentials::json_credentials() (gcsfs.h:47), which is not declared there today but is
already read by the R glue for its own options round-trip (r/src/filesystem.cpp:489-490).
Related: #34421 and #34524 (the R binding and its path-or-string handling), #33106 (its
docs), #32207 (the reverse gap, impersonation missing in R), #12763 (the original
[Python][C++] GCS Bindings, described there as "mostly based on AWS bindings"; I did not find
service-account JSON discussed on it), #11945 (the C++ credential types this builds on).
Does a constructor keyword fit how you would want this exposed, or would you rather it
arrive as a separate classmethod?
Component(s)
Python
Describe the enhancement requested
GcsOptions::FromServiceAccountCredentials(cpp/src/arrow/filesystem/gcsfs.h:146) is theone credential factory on
GcsOptionswith no Python binding, so a caller holdingaip/4112 service-account JSON has no way to construct a
GcsFileSystemfrom it.python/pyarrow/includes/libarrow_fs.pxddeclares the other four (Defaults,Anonymous,FromAccessToken,FromImpersonatedServiceAccount, lines 229-240), and thecredential-routing chain inside
GcsFileSystem.__init__(python/pyarrow/_gcsfs.pyx:103-131,method at line 88) selects among them. No constructor keyword reaches the fifth. Line
references are against
d048f71; behaviour checked on pyarrow 25.0.1.The R bindings already have it.
GcsFileSystem$create()takesjson_credentials, either theJSON itself or a path to a file holding it (
r/R/filesystem.R:611-674), and routes it toGcsOptions::FromServiceAccountCredentials(r/src/filesystem.cpp:415-417); it also rejectsjson_credentialsalongsideaccess_token, which is the mutual exclusion proposed below. Thatlanded in #34524 (GH-34421) in March 2023. So the two bindings have complementary holes: R has
service-account JSON and lacks impersonation (#32207), Python has impersonation
(
target_service_account) and lacks service-account JSON.Two routes are open today to an application holding that JSON in memory, and each gives
something up:
GOOGLE_APPLICATION_CREDENTIALSat it. That puts the privatekey on disk, and it configures the process rather than the filesystem object, so two
filesystems in one process cannot use different service accounts. The docstring on this
function names the case it exists for: "Using this function may be useful when the json
object is obtained from a Cloud Secret Manager or a similar service."
access_token+credential_token_expiration.Per
FromAccessToken's docstring those tokens are time limited and the caller refreshesthem manually, and the Python API offers no way to replace the token on an existing
GcsFileSystem, so a refresh means constructing a new one.Concrete case: omniload (panodata/omniload#233) accepts GCS service-account JSON as either a
path or a base64 blob from a secret store. Its S3 and R2 transports moved to
pyarrow.fsforthe per-file open latency (on that sibling transport, 8.68 ms/file through s3fs against
3.49 ms/file through Arrow, on the reader path that opens one file object per file). GCS
stayed on gcsfs, because the swap would drop that credential mode.
Proposed shape, following the existing chain:
FromServiceAccountCredentialsinlibarrow_fs.pxdbeside the other four.GcsFileSystem.__init__keyword taking the aip/4112 JSON as a string, in the samebase-credential branch as
anonymousandaccess_tokenand mutually exclusive with both.service_account_jsonreads more clearly thanservice_account_credentials, since thevalue is serialized JSON rather than a credentials object; the C++ accessor calls it
json_credentials. It would stay composable withtarget_service_accountby passing theresulting credentials to
FromImpersonatedServiceAccount, as the constructor already doesfor the other base credentials.
__reduce__round-tripsaccess_tokentoday (_gcsfs.pyx:183), but a long-lived private key is a differentproposition and declining to serialize this mode is a reasonable answer. If it should
round-trip through the existing
__reduce__design, the pxd would also needGcsCredentials::json_credentials()(gcsfs.h:47), which is not declared there today but isalready read by the R glue for its own options round-trip (
r/src/filesystem.cpp:489-490).Related: #34421 and #34524 (the R binding and its path-or-string handling), #33106 (its
docs), #32207 (the reverse gap, impersonation missing in R), #12763 (the original
[Python][C++] GCS Bindings, described there as "mostly based on AWS bindings"; I did not findservice-account JSON discussed on it), #11945 (the C++ credential types this builds on).
Does a constructor keyword fit how you would want this exposed, or would you rather it
arrive as a separate classmethod?
Component(s)
Python