-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make Helm artifacts reproducible #36930
Make Helm artifacts reproducible #36930
Conversation
5639244
to
f498fd4
Compare
f7a41ba
to
002000a
Compare
cc: @jedcunningham - I'd love you take a look before preparing the Helm Chart to see if that all looks good for you - I made all the variables and package location quite a bit more consistent. Also the source tarball contained postgres.tgz which I think it should not - users can download them on their own and we should not release those sources together with ours IMHO. |
002000a
to
30a692d
Compare
f8b8e74
to
04053a7
Compare
I actually managed to get also the helm package reproducible - It turned out to be as easy as repackaging the .tar.gz produced by helm-package in a reproducible way. Signing the packages with This way we can nicely combine reproducible packages, signing in ASF way and signing with .prov file. Pretty cool. |
fe47998
to
7532dc0
Compare
11697e9
to
0e4e6ec
Compare
dev/breeze/src/airflow_breeze/commands/release_management_commands.py
Outdated
Show resolved
Hide resolved
dev/breeze/src/airflow_breeze/commands/release_management_commands.py
Outdated
Show resolved
Hide resolved
830f08e
to
51b037b
Compare
Following apache#36726, apache#36744, apache#36763, apache#36819 this PR adds the feature of making source tarball that we release as an official release of the ASF for Helm Chart into reproducible tarball. This means that anyone should be able to produce such tarball using the sources of airflow and verify that he tarball pushed to SVN by the release manager is built from our source repositories. We also do the same with Helm package. It turns out that gpg signing of the package does not modify the .tgz file - it just adds .prov file containing checksum and signature, so we can safely re-pack the .tar.gz package in a reproducible way, this way we have both reproduciblity and provenance check nicely working together. There are few changes in this PR that are related: * Bumped Helm version in our environment to use the latest one and using the `breeze k8s setup-env` environment to run all the release commands - this way we can be sure same helm version is used to build the package, further making it more reproducible. * The reproducible packaging utility we have has been refeactored now - we take "source" archive as parameter rather than directory and simply repack it in reproducible way. * The tool also applies group/other ownership removal on its own, because helm package has no option to umask the generated files. * In this change we also ignore subcharts from being exported to the source tarball package as we shoudl not include source files from postgres in our source package.. * Both - the tarball and helm package are generated in `dist` folder similarly as all our other packages. * Documentation for releasing the packages and verifying them is updated. * CI jobs are updated to use the new commands and generated packages are produced as artifacts so that we can be sure the commands continue working and produce the right output.
51b037b
to
ed0584e
Compare
Following #36726, #36744, #36763, #36819 this PR adds the feature of making source tarball that we release as an official release of the ASF for Helm Chart into reproducible tarball. This means that anyone should be able to produce such tarball using the sources of airflow and verify that he tarball pushed to SVN by the release manager is built from our source repositories. We also do the same with Helm package. It turns out that gpg signing of the package does not modify the .tgz file - it just adds .prov file containing checksum and signature, so we can safely re-pack the .tar.gz package in a reproducible way, this way we have both reproduciblity and provenance check nicely working together. There are few changes in this PR that are related: * Bumped Helm version in our environment to use the latest one and using the `breeze k8s setup-env` environment to run all the release commands - this way we can be sure same helm version is used to build the package, further making it more reproducible. * The reproducible packaging utility we have has been refeactored now - we take "source" archive as parameter rather than directory and simply repack it in reproducible way. * The tool also applies group/other ownership removal on its own, because helm package has no option to umask the generated files. * In this change we also ignore subcharts from being exported to the source tarball package as we shoudl not include source files from postgres in our source package.. * Both - the tarball and helm package are generated in `dist` folder similarly as all our other packages. * Documentation for releasing the packages and verifying them is updated. * CI jobs are updated to use the new commands and generated packages are produced as artifacts so that we can be sure the commands continue working and produce the right output. (cherry picked from commit 48158c9)
Following #36726, #36744, #36763, #36819 this PR adds the feature of making source tarball that we release as an official release of the ASF for Helm Chart into reproducible tarball. This means that anyone should be able to produce such tarball using the sources of airflow and verify that he tarball pushed to SVN by the release manager is built from our source repositories. We also do the same with Helm package. It turns out that gpg signing of the package does not modify the .tgz file - it just adds .prov file containing checksum and signature, so we can safely re-pack the .tar.gz package in a reproducible way, this way we have both reproduciblity and provenance check nicely working together. There are few changes in this PR that are related: * Bumped Helm version in our environment to use the latest one and using the `breeze k8s setup-env` environment to run all the release commands - this way we can be sure same helm version is used to build the package, further making it more reproducible. * The reproducible packaging utility we have has been refeactored now - we take "source" archive as parameter rather than directory and simply repack it in reproducible way. * The tool also applies group/other ownership removal on its own, because helm package has no option to umask the generated files. * In this change we also ignore subcharts from being exported to the source tarball package as we shoudl not include source files from postgres in our source package.. * Both - the tarball and helm package are generated in `dist` folder similarly as all our other packages. * Documentation for releasing the packages and verifying them is updated. * CI jobs are updated to use the new commands and generated packages are produced as artifacts so that we can be sure the commands continue working and produce the right output. (cherry picked from commit 48158c9)
The apache#36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints
The apache#36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints
The apache#36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints
…43276) The #36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints
…pache#43276) The apache#36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints (cherry picked from commit 274b6e1)
#43298) * Fix edge-case when conflicting constraints prevent k8s env creation (#43276) The #36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints (cherry picked from commit 274b6e1) * Update k8s_requirements.txt
…pache#43276) The apache#36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints
…pache#43276) The apache#36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints
#43298) * Fix edge-case when conflicting constraints prevent k8s env creation (#43276) The #36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints (cherry picked from commit 274b6e1) * Update k8s_requirements.txt
…pache#43276) The apache#36930 added constraints to creation of k8s environment, but it had a side effect - the constraints could not be created if source of airflow had dependencies conflicting with constraints (which might happen for example when we upgrade FAB - because locally pinned FAB might be different than the one in constraints). Also the constraints were "hard-coded" - the python version, branch and github repository were hard-coded. This PR fixes both problems: * constraints URL is dynamically generated based on current branch, repo and python version * in case attempts to create the venv with constraints fails, we attempt to install it again without constraints
Following #36726, #36744, #36763, #36819 this PR adds the feature of
making source tarball that we release as an official release of
the ASF for Helm Chart into reproducible tarball. This means that
anyone should be able to produce such tarball using the sources
of airflow and verify that he tarball pushed to SVN by the
release manager is built from our source repositories.
We also do the same with Helm package. It turns out that gpg signing
of the package does not modify the .tgz file - it just adds .prov file
containing checksum and signature, so we can safely re-pack the .tar.gz
package in a reproducible way, this way we have both reproduciblity and
provenance check nicely working together.
There are few changes in this PR that are related:
Bumped Helm version in our environment to use the latest one and
using the
breeze k8s setup-env
environment to run all the releasecommands - this way we can be sure same helm version is used to build
the package, further making it more reproducible.
The reproducible packaging utility we have has been refeactored now -
we take "source" archive as parameter rather than directory and simply
repack it in reproducible way.
The tool also applies group/other ownership removal on its own,
because helm package has no option to umask the generated files.
In this change we also ignore subcharts from being exported to the source
tarball package as we shoudl not include source files from postgres in
our source package..
Both - the tarball and helm package are generated in
dist
folder similarly asall our other packages.
Documentation for releasing the packages and verifying them is updated.
CI jobs are updated to use the new commands and generated packages are
produced as artifacts so that we can be sure the commands continue
working and produce the right output.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.