Skip to content

Improve portability of dags deployer app #40

@jaketf

Description

@jaketf

Refactor gcloud composer environments run wrapper commands with something like kubectl exec.

Benefits

  1. generalize it's usefulness to "roll your own" k8s deployments or airflow.
  2. Pave the way to eliminate glcoud dependency of dagsdeployer app. (Would also require a refactor to use go storage client for file copying / hashing etc.) This would allow us to use a MUCH smaller container for the dagsdeployer app (e.g. distroless). However, this may be a lot of work for not much advantage.

Refactor description:

  • migrate most of the airflow logic out of composer_ops.go to a airflow_ops package
  • Add a k8s wrapper for airflow cli commands to replace the gcloud wrapper
  • Use the composer go client to retrieve k8s cluster details

Other considerations

This should include adding a k8s airflow deployment to the integration test infrastructure.

References:

k8s go client for kubectl exec from go code
composer go client (to retrieve k8s cluster details)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions