AWS EMR Operators: Add Log URI in task logs to speed up debugging

[akumar1-godaddy]

Description

Airflow is widely used to launch, interact and submit jobs on AWS EMR Clusters. Existing EMR operators do not provide links to the EMR logs (Job Flow/Step logs), as a result in case of failures the users need to switch to EMR Console or go to AWS S3 console to locate the logs for EMR Jobs and Steps using the job_flow_id available in the EMR Operators and in Xcom.

It will be really convenient and help with debugging if the EMR log links are present in Operator Task logs, it will obviate the need to switch to AWS S3 or AWS EMR consoles from Airflow and lookup the logs using job_flow_ids. It will be a nice improvement for the developer experience.

LogUri for Cluster is available in DescribeCluster

LogFile path for Steps in case of failure is available in ListSteps

Use case/motivation

Ability to go to EMR logs directly from Airflow EMR Task logs.

Related issues

N/A

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Isn't that done already via https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html ? There is EMR link already in the latest provider #28180 - maybe you are not using the latest provider simply? Or maybe it needs to be used in other operators?

Can you please @akumar1-godaddy

a) close the issue if it is already solved in the latest provider or
b) simply add PR to add them (following the one I copied)

[akumar1-godaddy]

Thanks @potiuk for the inputs and reference links.

I checked the EMR operators, as of now the EMR Operators only have links to the EMR Cluster Summary(EMR console) and the EMR Sensors have links to the S3 logs folder containing logs for nodes, steps and container.
I think adding the log uri to the operators would be helpful wherever applicable e.g. for failed steps it would be useful to have loguri for EmrAddStepsOperator.

Will work on the PR.

[shubham22]

+1 to suggestion from @akumar1-godaddy; having additional extra link should only improve the UX.