Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator

[olchas]

Description

GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator do not support direct impersonation of a service account in Google services.

Use case / motivation

GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator, in contrary to other Google operators, do not use Credentials class for authentication, so they require individual approach to support direct impersonation in them.

In case of GKEStartPodOperator it seems it should suffice to add --impersonate-service-account to the gcloud container clusters get-credentials command. However, this way we will not be able to use chain of service accounts, like in the rest of Google operators.

In case of DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator, some changes in provide_gcp_credential_file_as_context will probably be needed.

Related Issues

#8803

[rajatsri28]

Hi @olchas ,
Can I pick this up?

[mik-laj]

@rajatsri28 Do you have experience with Google Cloud? This task is trivial from an Airflow perspective, but requires above-average knowledge of the Google Cloud platform. I don't want you to have a bad experience after contributing to this project.

[rajatsri28]

Hi @mik-laj , Yes I have some experience with Google Cloud. At Twitter, we migrated from on-premise offering to provide Airflow on GKE (GCP) so I can somewhat relate to this issue as well.

[mik-laj]

@rajatsri28 Awesome! I assigned you to this ticket. 🐈

[rajatsri28]

Hi,
For GKEStartPodOperator, I did not find any way to use a chain of service accounts for impersonation. It seems it's only available via the API. So I am thinking of following what is suggested here, i.e. to add --impersonate-service-account to the gcloud container clusters get-credentials command.

For DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator, I am a bit confused since there are not many options to update ADC. Are we looking for something like running gcloud config set auth/impersonate_service_account command to declare the impersonation? However, we will not be able to specify a chain of accounts then.

[mik-laj]

@rajatsri28 I think support for a single service account is sufficient in most cases. gcloud config set auth/impersonate_service_account The gcloud command should work for all operators that use gcloud, so all you have to do is add account name forwarding in airflow.providers.google.common.hooks.base_google.GoogleBaseHook.provide_authorized_gcloud method.

gcloud config set auth/impersonate_service_account command and --impersonate-service-account argument behave similarly, but I think the first method will be simpler to implement. I am not sure about this and think it is worth checking for all the gcloud supported authentication methods.

[abhishekbafna]

The DataflowCreateJavaJobOperator is documented as deprecated and users are suggested to use the BeamRunJavaPipelineOperator. I tried that and I believe, I was not able to impersonate the different service account in that as well. The reason could be that the underline logic might be the same between the two operators. We should add these Beam operators also into the above list.