Support AWS Batch multinode job types

[camilleanne]

Description

Support multinode job types in the AWS Batch Operator.

The boto3 submit_job method supports container, multinode, and array batch jobs with the mutually exclusive nodeOverrides and containerOverrides (+ arrayProperties) parameters. But currently the AWS Batch Operator only supports submission of container jobs and array jobs by hardcoding the boto3 submit_job parameter containerOverrides:

airflow/airflow/providers/amazon/aws/operators/batch.py

Line 200 in 3c08cef

containerOverrides=self.overrides,

&

airflow/airflow/providers/amazon/aws/hooks/batch_client.py

Line 99 in 3c08cef

containerOverrides: Dict,

The get_job_awslogs_info method in the batch client hook is also hardcoded for the container type job:

airflow/airflow/providers/amazon/aws/hooks/batch_client.py

Line 425 in 3c08cef

job_container_desc = self.get_job_description(job_id=job_id).get("container", {})

To support multinode jobs the get_job_awslogs_info method would need to access nodeProperties from the describe_jobs response.

Use case/motivation

Multinode jobs are a supported job type of AWS Batch, are supported by the underlying boto3 library, and should be also be available to be managed by Airflow. I've extended the AWS Batch Operator for our own use cases, but would prefer to not maintain a separate operator.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Feel free. I assigned you. You might also sync this up with the work done by @ferruzzi @o-nikolas and @vincbeck from the AWS team -

[eladkal]

@camilleanne are you working on this issue?

[o-nikolas]

Unassigning Camille, so that others are free to take this task, since we have not heard back.

[o-nikolas]

@camilleanne has put out a PR (thanks Camille!) reassigning