[Python] Bindings for GcsFileSystem FromServiceAccountCredentials

[hampsterx]

Describe the enhancement requested

GcsOptions::FromServiceAccountCredentials (cpp/src/arrow/filesystem/gcsfs.h:146) is the
one credential factory on GcsOptions with no Python binding, so a caller holding
aip/4112 service-account JSON has no way to construct a GcsFileSystem from it.

python/pyarrow/includes/libarrow_fs.pxd declares the other four (Defaults, Anonymous,
FromAccessToken, FromImpersonatedServiceAccount, lines 229-240), and the
credential-routing chain inside GcsFileSystem.__init__ (python/pyarrow/_gcsfs.pyx:103-131,
method at line 88) selects among them. No constructor keyword reaches the fifth. Line
references are against d048f71; behaviour checked on pyarrow 25.0.1.

The R bindings already have it. GcsFileSystem$create() takes json_credentials, either the
JSON itself or a path to a file holding it (r/R/filesystem.R:611-674), and routes it to
GcsOptions::FromServiceAccountCredentials (r/src/filesystem.cpp:415-417); it also rejects
json_credentials alongside access_token, which is the mutual exclusion proposed below. That
landed in #34524 (GH-34421) in March 2023. So the two bindings have complementary holes: R has
service-account JSON and lacks impersonation (#32207), Python has impersonation
(target_service_account) and lacks service-account JSON.

Two routes are open today to an application holding that JSON in memory, and each gives
something up:

• Write it to a file and point GOOGLE_APPLICATION_CREDENTIALS at it. That puts the private
  key on disk, and it configures the process rather than the filesystem object, so two
  filesystems in one process cannot use different service accounts. The docstring on this
  function names the case it exists for: "Using this function may be useful when the json
  object is obtained from a Cloud Secret Manager or a similar service."
• Mint an access token out of band and pass access_token + credential_token_expiration.
  Per FromAccessToken's docstring those tokens are time limited and the caller refreshes
  them manually, and the Python API offers no way to replace the token on an existing
  GcsFileSystem, so a refresh means constructing a new one.

Concrete case: omniload (panodata/omniload#233) accepts GCS service-account JSON as either a
path or a base64 blob from a secret store. Its S3 and R2 transports moved to pyarrow.fs for
the per-file open latency (on that sibling transport, 8.68 ms/file through s3fs against
3.49 ms/file through Arrow, on the reader path that opens one file object per file). GCS
stayed on gcsfs, because the swap would drop that credential mode.

Proposed shape, following the existing chain:

• Declare FromServiceAccountCredentials in libarrow_fs.pxd beside the other four.
• Add a GcsFileSystem.__init__ keyword taking the aip/4112 JSON as a string, in the same
  base-credential branch as anonymous and access_token and mutually exclusive with both.
  service_account_json reads more clearly than service_account_credentials, since the
  value is serialized JSON rather than a credentials object; the C++ accessor calls it
  json_credentials. It would stay composable with target_service_account by passing the
  resulting credentials to FromImpersonatedServiceAccount, as the constructor already does
  for the other base credentials.
• Pickling would need a policy decision, and I have not assumed one. __reduce__ round-trips
  access_token today (_gcsfs.pyx:183), but a long-lived private key is a different
  proposition and declining to serialize this mode is a reasonable answer. If it should
  round-trip through the existing __reduce__ design, the pxd would also need
  GcsCredentials::json_credentials() (gcsfs.h:47), which is not declared there today but is
  already read by the R glue for its own options round-trip (r/src/filesystem.cpp:489-490).

Related: #34421 and #34524 (the R binding and its path-or-string handling), #33106 (its
docs), #32207 (the reverse gap, impersonation missing in R), #12763 (the original
[Python][C++] GCS Bindings, described there as "mostly based on AWS bindings"; I did not find
service-account JSON discussed on it), #11945 (the C++ credential types this builds on).

Does a constructor keyword fit how you would want this exposed, or would you rather it
arrive as a separate classmethod?

Component(s)

Python