-
Notifications
You must be signed in to change notification settings - Fork 10
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Refactor gcloud composer environments run wrapper commands with something like kubectl exec.
Benefits
- generalize it's usefulness to "roll your own" k8s deployments or airflow.
- Pave the way to eliminate glcoud dependency of dagsdeployer app. (Would also require a refactor to use go storage client for file copying / hashing etc.) This would allow us to use a MUCH smaller container for the dagsdeployer app (e.g. distroless). However, this may be a lot of work for not much advantage.
Refactor description:
- migrate most of the airflow logic out of composer_ops.go to a airflow_ops package
- Add a k8s wrapper for airflow cli commands to replace the gcloud wrapper
- Use the composer go client to retrieve k8s cluster details
Other considerations
This should include adding a k8s airflow deployment to the integration test infrastructure.
References:
k8s go client for kubectl exec from go code
composer go client (to retrieve k8s cluster details)
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request