Using Application Default Credentials lead to an error

[dbalabka]

Describe the issue:

GCP documentation provides the following recommendation about local development and application authentification:

When your code is running in a local development environment, such as a development workstation, the best option is to use the credentials associated with your user account.

Simply saying we have to generate the credential json file using the following command:

gcloud auth application-default login

which will be located in ~/.config/gcloud/application_default_credentials.json. The following JSON credential file is different to service account file and does not contain some certaint fields like client_email. However, it is fully legit to specify this file in GOOGLE_APPLICATION_CREDENTIALS. The following env variable can contain a path to three types of JSON file: Workforce Identity Federation, Workload Identity Federation or service account key.
Currently, dask-cloudprovider expects that GOOGLE_APPLICATION_CREDENTIALS can contain only service account key and using Workload Identity Federation key leads to an error.

Here are a steps to reproduce the problem:

1. Create ~/.config/gcloud/application_default_credentials.json by executing the command:

gcloud auth application-default login

2. Set the absolute path of ~/.config/gcloud/application_default_credentials.json into GOOGLE_APPLICATION_CREDENTIALS variable.
3. Try to start cluster on GCP

ER: it should work
AR: we get an error of incorrect format service account key.

I propose to change the way how Dask obtain credentials by replacing the existing logic with default Google's SDK flow of obtaining the application default credentials unless there is an importnat reason to leave the current logic.

The default Google's flow is the very close to which is implemented in Dask:

ADC searches for credentials in the following locations:

1. GOOGLE_APPLICATION_CREDENTIALS environment variable
2. User credentials set up by using the Google Cloud CLI
3. The attached service account, returned by the metadata server

Link to PR: #430

Environment:

• Dask version:

dask = "2024.5.2"
dask-cloudprovider = {extras = ["gcp"], version = "^2022.10.0"}

• Python version: 3.10
• Operating System: WSL
• Install method (conda, pip, source): poetry

[dbalabka]

@jacobtomlinson, the following ticket can be closed. I'm using the latest version of dask-cloudprovider===2024.9.1 that already provides #430 changes.