"AirflowException: Invalid arguments were passed to GlueJobOperator" when setting update_config=True

[andrewparsons-janus]

Apache Airflow version

Other Airflow 2 version (please specify below)

Airflow version: 2.4.3
apache-airflow-providers-amazon==8.7.1

What happened

Exception when loading DAG:

Broken DAG: [/opt/airflow/dags/<project_directory>/<dag>.py] Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 408, in apply_defaults
    result = func(self, **kwargs, default_args=default_args)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 756, in __init__
    raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to GlueJobOperator (task_id: submit_glue_job). Invalid arguments were:
**kwargs: {'update_config': True}

What you think should happen instead

I don't believe passing an argument to GlueJobOperator's update_config init parameter should throw an exception. I also don't believe that I should have to set ALLOW_ILLEGAL_ARGUMENTS. Why would a documented parameter be illegal?

If this is expected behavior, then it should be better documented.

How to reproduce

If needed, I can provide a comprehensive example, but anyone should be able to adapt this task with a basic DAG.

@task_group(group_id="...", default_args=None)
def run_glue_job(job_name: str, script_args: dict):
    submit_glue_job = GlueJobOperator(

        # BaseOperator kwargs
        task_id="submit_glue_job",
        retries=0,
        wait_for_completion=False,
        on_success_callback=foo,
        on_failure_callback=[foo, bar],

        # GlueJobOperator kwargs
        job_name=job_name,
        script_location=f"{S3_PATH/glue_script.py",
        script_args=script_args,

        update_config=True,  # <-- this causes an exception!

        **EXTRA_KWARGS_GLUE,
    )

    wait_on_glue_job = GlueJobSensor(
        task_id="wait_on_glue_job",
        job_name=job_name,
        run_id=submit_glue_job.output,  # type: ignore
        on_failure_callback=bar,
    )

    submit_glue_job >> wait_on_glue_job

Operating System

macOS 14.1.1

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.7.1

Deployment

Docker-Compose

Deployment details

Docker version 20.10.24, build 297e128

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

---

Edit 1: adjust listed version for apache-airflow-providers-amazon

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

Unable to reproduce on latest main.

This DAG created fine without any issues

import pendulum

from airflow.decorators import task_group
from airflow.models.dag import DAG
from airflow.providers.amazon.aws.sensors.glue import GlueJobSensor
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator


S3_PATH = "s3://foo/bar"
EXTRA_KWARGS_GLUE = {}


with DAG(
    "issue_35637",
    start_date=pendulum.datetime(2023, 6, 1, tz="UTC"),
    schedule=None,
    catchup=False,
    tags=["issue", "35637"]
):

    @task_group(group_id="foo-bar", default_args=None)
    def run_glue_job(job_name: str, script_args: dict):
        submit_glue_job = GlueJobOperator(
            # BaseOperator kwargs
            task_id="submit_glue_job",
            retries=0,
            wait_for_completion=False,
            # GlueJobOperator kwargs
            job_name=job_name,
            script_location=f"{S3_PATH}/glue_script.py",
            script_args=script_args,
            update_config=True,
            **EXTRA_KWARGS_GLUE,
        )

        wait_on_glue_job = GlueJobSensor(
            task_id="wait_on_glue_job",
            job_name=job_name,
            run_id=submit_glue_job.output,
        )

        submit_glue_job >> wait_on_glue_job


    run_glue_job(job_name="foo", script_args={})

You need to check on more recent version of Airflow, there is such of changes and fixes in Airflow Core since then, include task_group decorator.

In additional you need to check that all of your Airflow components use same amazon-provider. The changes which add this field is 7.4.0 (#30162) and maybe your scheduler/dag_processor/worker use different version (6.0.0 ???) which are below of this version

[github-actions]

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[andrewparsons-janus]

This was embarrassing. Sorry for creating noise! I was indeed running an old version of apache-airflow-providers-amazon without realizing it.