Add support for Databricks oauth federation auth

[devlucasc]

Description

Databricks has release as Public Preview the OIDC Federation authentication that allows to authenticate without need of rotating the service principal secret. The details can be found here: https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation-provider

Use case/motivation

Using this type of authentication is more secure because it eliminates the need to manage credentials and can be restricted to an AWS role, for example. It also makes the Airflow instance more resilient and reliable, preventing pipeline failures when a token expires—such as when a user forgets to rotate the secret or when an automated rotation process fails.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[devlucasc]

I could find this implementation on how Databricks Python SDK works https://github.com/databricks/databricks-sdk-py/blob/main/databricks/sdk/oidc.py#L19

[mwojtyczka]

I'm working on adding support for OIDC token federation to the Databricks provider. Starting with Token Federation for Airflow clusters running on Kubernetes. Next support for OIDC token federation for cluster deployed on AWS using STS Federation which has recently be added: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_outbound_getting_started.html
I expect to complete this in few weeks.