Add extra links for EMR Serverless

[dacort]

Description

Add extra links to the EMR Serverless operator for S3 logs as well as the GetDashboardForJobRun API that returns a one-time link to the UI.

Use case/motivation

Allow for easier debugging of EMR Serverless jobs. Today, folks have to navigate through the AWS console to find the corresponding application and job run ID in order to gain access to the Spark UI or stdout/stderr logs. It'd be great to have direct links for both of those things.

One potential challenge is that the URL returned by the GetDashboardForJobRun API is both short-lived (expires after 1 hour) and one-time (expires on first use). I'm concerned about both poor user experience of clicking an expired or already used URL and the API impact of generating a URL every time the DAG popup is triggered even if it's not used. This may require some additional functionality in Airflow to implement.

Related issues

#29939

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[o-nikolas]

Thanks! I don't worry too much about the overhead of generating the link each time the dag popup loads, but I 1000% agree with the poor user experience, we need to have an answer for that.

I like the idea of a new class of extra links. Instead of a link it would be a button that says something like "generate link to " and then upon clicking that some client side code runs to generate the link (the user sees a spinner while that's happening but it should be fast) and then they are presented with the link once it's ready.

[phanikumv]

@dacort assigning it to you as you mentioned that you are willing to submit a PR for this

[dacort]

Thanks @phanikumv - I do have an early branch ready, but I still need to figure out how to handle the one-time URL generated by the GetDashboardForJobRun API.

My current thought is to add new endpoint (something like /extra_link_ephemeral) that generates the link and redirects the user, but this is a bit more of a core change to Airflow.

[eladkal]

Hi @dacort are you still working on this issue?

[dacort]

I am yes, hope to push something up early next week.

• Don't think I will try to figure out a new approach for ephemeral links. Some changes were made on EMR side to make the user experience better
• Also considering add more links (like to S3 and/or CloudWatch, but then the links started going off the pop up dialog. :/