Switches to Helm Chart for Kubernetes tests #9468
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
force-pushed
the
switch-kubernetes-tests-to-helm-chart
branch
2 times, most recently
from
4aa3182 to
1466cc2
Compare
|
Hey @dimberman @ashb @kaxil -> I managed to get all the kubernetes tests working using the helm chart :) (with lots of @dimberman's help) . I got rid of all the "templates" from the old tests. The problems that prevented the tests to run turned to be simple: missing custom "resource + resource claim" for two of the tests and the fact that it takes a few seconds for the webserver to start and I had to add a loop to test if it is up when I setup port-forwarding. That seems to fix all the problems - and we can start adding more tests for kubernetes installation now. I also added more logs for the tests and breeze's "kind-cluster shell' command that drops you to shell where you can run kubernetes tests... Looking forward to merging it (I will cherry-pick to 1.10 right after) |
potiuk
commented
Jun 25, 2020
potiuk
force-pushed
the
switch-kubernetes-tests-to-helm-chart
branch
2 times, most recently
from
2e6978f to
9b0d480
Compare
potiuk
force-pushed
the
switch-kubernetes-tests-to-helm-chart
branch
2 times, most recently
from
e3ff9fe to
c3b7974
Compare
potiuk
changed the title
potiuk
force-pushed
the
switch-kubernetes-tests-to-helm-chart
branch
5 times, most recently
from
13c95dc to
cba3e31
Compare
|
I believe this one should succeed in the CI as well as locally with Breeze and it will make it much easier to iterate on future changes to both - image and helm chart! |
mik-laj
reviewed
Jun 28, 2020
mik-laj
reviewed
Jun 28, 2020
| @@ -108,6 +108,7 @@ def start_pod( | |||
| curr_time = dt.now() | |||
| if resp.status.start_time is None: | |||
| while self.pod_not_started(pod): | |||
| self.log.warning("Pod not yet started: %s", pod.metadata.name) | |||
There was a problem hiding this comment.
Should we remove log print on 116?
potiuk
force-pushed
the
switch-kubernetes-tests-to-helm-chart
branch
3 times, most recently
from
a04338d to
2199d3c
Compare
dimberman
approved these changes
Jun 30, 2020
|
@potiuk CI passed :) |
6 tasks
The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io>
potiuk
force-pushed
the
switch-kubernetes-tests-to-helm-chart
branch
from
2199d3c to
962865b
Compare
potiuk
added a commit
that referenced
this pull request
Jul 1, 2020
The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io> (cherry picked from commit 8bd15ef)
potiuk
added a commit
that referenced
this pull request
Jul 1, 2020
kaxil
pushed a commit
that referenced
this pull request
Jul 1, 2020
kaxil
pushed a commit
that referenced
this pull request
Jul 2, 2020
kaxil
pushed a commit
that referenced
this pull request
Jul 2, 2020
The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io> (cherry picked from commit 8bd15ef)
kaxil
pushed a commit
that referenced
this pull request
Jul 13, 2020
The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io> (cherry picked from commit 8bd15ef)
13 tasks
cfei18
pushed a commit
to cfei18/incubator-airflow
that referenced
this pull request
Mar 5, 2021
cfei18
pushed a commit
to cfei18/incubator-airflow
that referenced
this pull request
Mar 5, 2021
The Kubernetes tests are now run using Helm chart rather than the custom templates we used to have. The Helm Chart uses locally build production image so the tests are testing not only Airflow but also Helm Chart and a Production image - all at the same time. Later on we will add more tests covering more functionalities of both Helm Chart and Production Image. This is the first step to get all of those bundle together and become testable. This change introduces also 'shell' sub-command for Breeze's kind-cluster command and EMBEDDED_DAGS build args for production image - both of them useful to run the Kubernetes tests more easily - without building two images and with an easy-to-iterate-over-tests shell command - which works without any other development environment. Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Daniel Imberman <daniel@astronomer.io> (cherry picked from commit 8bd15ef)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switches to Helm Chart for Kubernetes tests
The Kubernetes tests are now run using Helm chart
rather than the custom templates we used to have.
The Helm Chart uses locally build production image
so the tests are testing not only Airflow but also
Helm Chart and a Production image - all at the
same time. Later on we will add more tests
covering more functionalities of both Helm Chart
and Production Image. This is the first step to
get all of those bundle together and become
testable.
This change introduces also 'shell' sub-command
for Breeze's kind-cluster command and
EMBEDDED_DAGS build args for production image -
both of them useful to run the Kubernetes tests
more easily - without building two images
and with an easy-to-iterate-over-tests
shell command - which works without any
other development environment.
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.