GlueJobOperator throws error after migration to newest version of Airflow

[vgutkovsk]

Apache Airflow version

2.5.1

What happened

We were using GlueJobOperator with Airflow 2.3.3 (official docker image) and it was working well, we didn't specify script file location, because it was inferred from the job name. After migration to 2.5.1 (official docker image) the operator fails if s3_bucket and script_location are not specified. That's the error I see:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 146, in execute
    glue_job_run = glue_job.initialize_job(self.script_args, self.run_job_kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 155, in initialize_job
    job_name = self.create_or_update_glue_job()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 300, in create_or_update_glue_job
    config = self.create_glue_job_config()
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 97, in create_glue_job_config
    raise ValueError("Could not initialize glue job, error: Specify Parameter `s3_bucket`")
ValueError: Could not initialize glue job, error: Specify Parameter `s3_bucket`

What you think should happen instead

I was expecting that after migration the operator would work the same way.

How to reproduce

Create a dag with GlueJobOperator operator and do not use s3_bucket or script_location arguments

Operating System

Linux

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==7.1.0

Deployment

Docker-Compose

Deployment details

apache/airflow:2.5.1-python3.10 Docker image and official docker compose

Anything else

I believe it was commit #27893 by @romibuzi that introduced this behaviour.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[romibuzi]

Hi @vgutkovsk!

Oh damn indeed I realize introduced a breaking change. Before the check if self.s3_bucket is None was done only when the operator was creating the job. Now it is done at the start of create_glue_job_config() method here:

airflow/airflow/providers/amazon/aws/hooks/glue.py

Lines 103 to 104 in 4402456

	if self.s3_bucket is None:
	raise ValueError("Could not initialize glue job, error: Specify Parameter `s3_bucket`")

And this method is called in any cases here:

airflow/airflow/providers/amazon/aws/hooks/glue.py

Line 328 in 4402456

config = self.create_glue_job_config()

I realize s3_bucket is only used to determine s3_log_path:

airflow/airflow/providers/amazon/aws/hooks/glue.py

Line 112 in 4402456

s3_log_path = f"s3://{self.s3_bucket}/{self.s3_glue_logs}{self.job_name}"

script_location on the other hand can be None and is not concatenated with s3_bucket at all.

Maybe the best way to handle the problem would be to remove this check on s3_bucket, and if it is None then omit the the parameter "LogUri" which makes usage of s3_log_path as it is not a mandatory parameter for a glue job:

airflow/airflow/providers/amazon/aws/hooks/glue.py

Line 118 in 4402456

"LogUri": s3_log_path,

cc @Taragolis

[Taragolis]

Oh damn indeed I realize introduced a breaking change.

¯\_(ツ)_/¯ this things sometimes happen, but I guess we could fix it by your proposal. Would you like to work on this issue?

[romibuzi]

@Taragolis Yeah I can work on the fix, I also saw that @vgutkovsk is willing to submit a PR so as you want :)

[gabFirmaway]

Oh! I was stuck with this today too!
I tested the triggering of existing Glue jobs on my development machine with an older version of Airflow and everything worked flawlessly.
Today, when I create an ec2 instance with a custom airflow docker container including these libraries and triggered the job, it keeps asking for s3_bucket. And if you provide the s3_bucket, it override the configuration that you create in the Glue Editor because it creates a new job overwriting the existing one in AWS (for example, for python only scripts, it changes to GlueVersion 3.0 but without defining the language, in this case, Python).

A temporary solution would be to add in the task definition:

create_job_kwargs={'GlueVersion':'3.0', "DefaultArguments": {"--job-language": "python"}}

But well, I guess that we need a more permanent fix to avoid overwriting the existing AWS Glue Jobs created.

[Taragolis]

I've never like design of GlueJobOperator (previously AwsGlueJobOperator) because a lot of different things combine in one single operator: Upload Artifact to S3, Create/Update Job, Run Job.

Seems like it should be decompose on different Operators, when I used Glue as service (hopefully not use it anymore) I've create and update it outside of Airflow as part of the CI process, so for me only Run Job was useful part but someone required also create/update it in Airflow in this case could use multiple chained operators.

But for now would be nice if someone create a fix for current operator. Just let know who want to make a PR soon for avoid situation that multiple people work on the same issue 😉

[romibuzi]

@Taragolis I agree, i can definitely start working on the fix as it was my previous contribution which leads to the current issue

[romibuzi]

@gabFirmaway Which parameters did you set in create_job_kwargs of your task definition?

The GlueJobOperator should only update parameters which are set in the DAG task definition in case there is something different defined in Glue:

airflow/airflow/providers/amazon/aws/hooks/glue.py

Lines 307 to 309 in fdac67b

	update_config = {
	key: value for key, value in job_kwargs.items() if current_job.get(key) != job_kwargs[key]
	}