Allow templating for more arguments in RunPipelineJobOperator

[ferenc-hechler]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

No response

Apache Airflow version

2.10.2

Operating System

GCP Composer 2.9.11

Deployment

Google Cloud Composer

Deployment details

No response

What happened

I tried to use jinja templating to set parameters like "template_path", "pipeline_root" or "service_account".
The templates are not replaced with actual values, instead of service_account is set to the string {{ var.value... }}.

What you think should happen instead

templates should also be supported for these fields and the value of the environment variable should be given.

How to reproduce

Executing the following task fails:

    run_task = RunPipelineJobOperator(
        task_id="run_pipeline_job",
        display_name="Run Pipeline Job",
        template_path="{{ var.value.KFP_TEMPLATE_URL }}",
        parameter_values={},
        region="europe-west3",
        project_id="my-project-id",
    )

The AirFlow variable KFP_TEMPLATE_URL is set to a valid template uploaded to an Kubeflow Pipeline Artifact Registry.
In the error message it is to see, that template_path was not resolved.

invalid URL "{{ var.value.KFP_TEMPLATE_URL }}"

Anything else

I was able to activate templating by inheriting the RunPipelineJobOperator and extend the class variable tempalte_fields:

airflow/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py

Lines 110 to 115 in e9c49d6

	template_fields = [
	"region",
	"project_id",
	"input_artifacts",
	"impersonation_chain",
	]

    template_fields = [
        "region",
        "project_id",
        "input_artifacts",
        "impersonation_chain",
[+]     "template_path",
[+]     "pipeline_root",
[+]     "parameter_values",
[+]     "service_account",
    ]

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[ferenc-hechler]

There exists also a discussion for this: #48627

[gyli]

I don't see a problem adding these fields to template fields. @MaksYermak do you remember why they were not template field at the beginning?

[MaksYermak]

I don't see a problem adding these fields to template fields. @MaksYermak do you remember why they were not template field at the beginning?

At the beginning we, always, don't know what fields users want to have as templated. We add only the most useful to template_fields and then extended it. I, also, do not see any problems with these new fields.

[gyli]

Hi @ferenc-hechler shall I assign this issue to you if you want to create a PR for this?

[ferenc-hechler]

There is already a PR for this: #50220