Add third party license + source code to argo and minio images to comply with their license #2201
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # How to release third party images | ||
|
|
||
| ## Release a new third party image version | ||
|
|
||
| 1. Edit third_party/$LIBRARY/Dockerfile | ||
| 1. Change the line `from IMAGE_NAME:TAG_NAME` to `from IMAGE_NAME:NEW_TAG_NAME` | ||
| 1. Edit third_party/$LIBRARY/release.sh | ||
| 1. Change TAG to NEW_TAG_NAME. | ||
| 1. Commit and ask someone for review | ||
| 1. Run the following (you need to have storage access to ml-pipeline project) | ||
| ``` | ||
| cd $KFP_SRC | ||
| ./third_party/$LIBRARY/release.sh | ||
| ``` | ||
| 1. Make a PR that | ||
| * changes all image references in .release.cloudbuild.yaml | ||
| * changes all image references in manifests | ||
|
|
||
| ## How to build | ||
| ``` | ||
| cd $KFP_SRC | ||
| gcloud builds submit --config third_party/argo/cloudbuild.yaml . --substitutions=TAG_NAME="RELEASE_TAG_NAME_HERE" | ||
| gcloud builds submit --config third_party/minio/cloudbuild.yaml . --substitutions=TAG_NAME="RELEASE_TAG_NAME_HERE" | ||
| ``` | ||
|
|
||
| or you can build locally using docker too like the following | ||
| ``` | ||
| cd $KFP_SRC | ||
| docker build -t $IMAGE_NAME:$TAG -f third_party/minio/Dockerfile . | ||
| ``` | ||
|
|
||
| ## Verify your built images are good | ||
| Run the following command to start an interactive shell in a new container of the image (the image must have shell installed to be able to run it) | ||
| ``` | ||
| docker run -it --entrypoint sh $IMAGE_NAME | ||
| ``` | ||
| Or if the image doesn't have a complete OS (like argoproj/workflow-controller) | ||
| ``` | ||
| docker save nginx > nginx.tar | ||
| tar -xvf nginx.tar | ||
| ``` | ||
| This saves layers of the image to a tarball that you can extract and see. | ||
|
|
||
| Credits to: https://stackoverflow.com/questions/44769315/how-to-see-docker-image-contents | ||
|
|
||
| ## Release to gcr.io/ml-pipeline | ||
|
|
||
| (This has been automated by third_party/release.sh) | ||
| 1. First build images in your own project | ||
| 1. Use [this gcloud command](https://cloud.google.com/container-registry/docs/managing#tagging_images) to retag your images to gcr.io/ml-pipeline | ||
| 1. When choosing the new tag, use the same text as the original release tag of the third party image |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # Copyright 2019 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| FROM alpine as downloader | ||
|
|
||
| COPY third_party /third_party | ||
|
|
||
| RUN apk add --no-cache curl | ||
|
|
||
| ENV OUTPUT_FOLDER /downloads/ | ||
| RUN sh /third_party/download_source.sh </third_party/argo/repo-MPL.txt | ||
|
|
||
|
|
||
| # Argo exec image | ||
| From argoproj/argoexec:v2.3.0 | ||
|
|
||
| # Copy concatenated license file | ||
| COPY third_party/argo/license.txt /third_party/license.txt | ||
|
|
||
| # Copy downloaded source code | ||
| COPY --from=downloader /downloads /third_party |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Copyright 2019 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| FROM alpine as downloader | ||
|
|
||
| COPY third_party /third_party | ||
|
|
||
| RUN apk add --no-cache curl | ||
|
|
||
| ENV OUTPUT_FOLDER /downloads/ | ||
| RUN sh /third_party/download_source.sh </third_party/argo/repo-MPL.txt | ||
|
|
||
| From argoproj/workflow-controller:v2.3.0 | ||
|
|
||
| # Copy concatenated license file | ||
| COPY third_party/argo/license.txt /third_party/license.txt | ||
|
|
||
| # Copy downloaded source code | ||
| COPY --from=downloader /downloads /third_party |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| steps: | ||
| - name: 'gcr.io/cloud-builders/docker' | ||
| args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/workflow-controller:$TAG_NAME', '-f', 'third_party/argo/Dockerfile.workflow-controller', '.' ] | ||
| waitFor: ['-'] | ||
| - name: 'gcr.io/cloud-builders/docker' | ||
| args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/argoexec:$TAG_NAME', '-f', 'third_party/argo/Dockerfile.argoexec', '.' ] | ||
| waitFor: ['-'] | ||
| timeout: 300s # 5min | ||
| options: | ||
| diskSizeGb: 20 | ||
| images: | ||
| - 'gcr.io/$PROJECT_ID/argoexec' | ||
| - 'gcr.io/$PROJECT_ID/workflow-controller' |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's move all these entries to a separate cloudbuild.yaml file in the /third-party directory so that it's not triggered with every release.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't quite understand why we have these entries. They are copying already built images from gcr.io/ml-pipeline/... to gcr.io/ml-pipeline/google/pipelines/...:$TAG_NAME (released tag name).
But I couldn't find any usage of gcr.io/ml-pipeline/google/pipelines, our lite deployment only references gcr.io/ml-pipeline/image-name.
Can you explain what these are for? I didn't understand so I merely changed the tags to my new tags.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also checked https://github.com/kubeflow/manifests, they are also not using these. @IronPan do you know what these are for? Just meant as taking a snapshot of the release artifacts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Bobgy no need to add license-compliance part in the tag here. As long as we're building these default ones with the source and license included, we're good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@neuromage My reasoning of the extra
license-compliancepart: #2201 (comment)It's for purpose of safe gradual release, because images like
gcr.io/ml-pipeline/argoexec:v2.3.0are already in use today. If I push a new image to the same tag, everyone will switch to that new image immediately.Technically, there's no difference between previous and new images. New images also just passed our presubmit tests. So risk pushing new images under the same tag is small too. I think we can also choose to do that.
WDYT? I'd like to double check if you want me to push new images under the same tag? or would you want to suggest a simpler naming?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think if we're distributing the images, we should just version them (since we technically rebuild these every release). Ideally, we would name them something like
argoexec-v2.3.0-kfp:0.1.30which can indicate that this is based of argo v2.3.0, but released under kfp with tag 0.1.30. I don't want to block this PR, but can we consider using this convention? This can be done in a future PR./cc @IronPan
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using different tags for the same image might be a bit confusing.