Improve production image iteration speed #9162
Conversation
|
Hey @ashb @dimberman @schnie @jhtimmins I think you might want to merge that one :). It is on the biggish side and I can split it into few independent PRs if you think that it makes sense, but I think this one is crucial to get fast iteration on the production image while we are integrating the helm chart with it. Yesterday when @ashb worked on the chart we had slack conversation and I could help fairly quickly in adding some changes to the entrypoint, i saw later @ashb mentioned lack of clear-logs as another blocker. So I realised we need to have a way so that not only me but all of us can actually build and release new images quickly. I am happy to help (and I have a bunch of changes in the GitHub issues but since we work sometimes in different timezones, I think it is important to get iterations quicker. This change implements a number of related improvements to Breeze to make it super easy to iterat with production images - including production images from released version of Airflow - and allows to quickly release patched versions. I already released -1 yesterday for @ash and I am just pushing -2 with clear-logs. You can take a look at the IMAGES.rst changes where I described everything but for your use it should be as easy as:
That's it. This will build production image out of PyPi 1.10.10 with the local Dockerfile and scripts - so you can easily add new scripts, use them in Dockerfile and build production image locally. Then you can re-tag and push them manually. What's super important - the PROD image is now optimized for BOTH - speed of rebuilds AND size. If you use the There are more changes in there for example once you build the prod image you can enter:
And few more things that I described in the description of the commit. I also added "1.10.10-1, -2" "release notes as well in the IMAGES.rst so that we can keep track of the changes while we work on them. |
potiuk
force-pushed
the
standardize-image-parameter-passing
branch
4 times, most recently
from
194354c to
435ae9b
Compare
|
I would love to get that one merged so that we can improve the prod images faster |
What do you think about entrypoint.sh → scripts/prod/entrypoint.sh |
I prefer to keep the name _prod - we also have a _ci one and _exec,sh. It's very easy to make a mistake which one you are editing (happened to me already several times). Adding explicit name makes it straightforward. |
| exec /bin/bash -c "$(printf "%q " "${ARGS[@]}")" | ||
| fi | ||
| if [[ "${RUN_TESTS:=false}" != "true" ]]; then | ||
| exec /bin/bash "${@}" |
There was a problem hiding this comment.
Are you sure of it? Should not be added --c here?
There was a problem hiding this comment.
Yep. I am sure. We want to add -c manually if needed. If you do not add anything you are simply dropped into shell. This way you can also pass additional options to bash - not only -c.
See the examples in BREEZE.rst
There was a problem hiding this comment.
This way you can do "./breeze shell -- -c "ls" or "./docker run IMAGE bash -c "ls" which is exactly the way how Bash's docker is used.
There was a problem hiding this comment.
Right. I think it's not needed after recent HIVE mocking. Let's see what our tests have to say about it.
|
It still feels very weird that we are building 1.10.10 images not from the Dockerfile in the 1.10.10 branch. (It makes reproducing/knowing what exactly went in to a tagged docker image harder.) |
potiuk
force-pushed
the
standardize-image-parameter-passing
branch
from
a8a9943 to
e0d5077
Compare
This is mainly for iterating now with the image. I am (as we speak) bringing the same Docker image changes to v1-10-test and we will build it from sources in 1.10.11 once we implemented everything we need in 1.10.11. This way we can build released 1.10.10 + latest Dockerfile But for the official releases it will be all from the same tag. |
potiuk
force-pushed
the
standardize-image-parameter-passing
branch
from
e0d5077 to
fd27d3b
Compare
|
Hey @ash -I am just about to finish cherry-picking to v1-10-test - with that one merged/cherry-picked it will be also possible to build the same improved image from current v1-10-test/stable directly. |
|
Guys. This is very annoying to get email notifications when you mistype the name of one of you. |
|
Hey @ashb -> Any more comments ? or ok to merge (would love to cherry-pick that one now as well) |
potiuk
force-pushed
the
standardize-image-parameter-passing
branch
from
fd27d3b to
1028e4f
Compare
potiuk
force-pushed
the
standardize-image-parameter-passing
branch
from
1028e4f to
05af850
Compare
For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
potiuk
force-pushed
the
standardize-image-parameter-passing
branch
from
05af850 to
815bc37
Compare
|
Only Quarantine failed. If there are no more comments I am emerging this one shortly @feluelle - you wanted to take a look as well ? |
For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images. (cherry picked from commit 7c12a9d)
For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images.
For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images. (cherry picked from commit 7c12a9d)
For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images. (cherry picked from commit 7c12a9d)
For a long time the way how entrypoint worked in ci scripts was wrong. The way it worked was convoluted and short of black magic. This did not allow to pass multiple test targets and required separate execute command scripts in Breeze. This is all now straightened out and both production and CI image are always using the right entrypoint by default and we can simply pass parameters to the image as usual without escaping strings. This also allowed to remove some breeze commands and change names of several flags in Breeze to make them more meaningful. Both CI and PROD image have now embedded scripts for log cleaning. History of image releases is added for 1.10.10-* alpha quality images. (cherry picked from commit 7c12a9d)
For a long time the way how entrypoint worked in ci scripts
was wrong. The way it worked was convoluted and short of black
magic. This did not allow to pass multiple test targets and
required separate execute command scripts in Breeze.
This is all now straightened out and both production and
CI image are always using the right entrypoint by default
and we can simply pass parameters to the image as usual without
escaping strings.
This also allowed to remove some breeze commands and
change names of several flags in Breeze to make them more
meaningful.
Both CI and PROD image have now embedded scripts for log
cleaning.
History of image releases is added for 1.10.10-*
alpha quality images.
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.