Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgraded Argo to v2.7.5 #3537

Merged
merged 6 commits into from
May 12, 2020
Merged

Upgraded Argo to v2.7.5 #3537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
411ba9c
Upgraded Argo to v2.7.4
Ark-kun Apr 17, 2020
c7d31d1
Downgraded the Argo CLI version to 2.4.3
Ark-kun Apr 24, 2020
716d29e
Removed the argo cli arg that had been removed
Ark-kun Apr 24, 2020
273ffd5
Updated to Argo 2.7.5
Ark-kun Apr 28, 2020
ed99b48
Added workflowtemplates and cronworkflows to the Role
Ark-kun Apr 28, 2020
6bbf789
Added the new Argo CRDs
Ark-kun Apr 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .cloudbuild.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,10 +211,10 @@ steps:
args: ['pull', 'gcr.io/cloudsql-docker/gce-proxy:1.14']
id: 'pullCloudsqlProxy'
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, the suffix -license-compliance was added to distinguish it from gcr.io/ml-pipeline/argoexec:v2.3.0, there's no longer such a need because gcr.io/ml-pipeline/argoexec:v2.7.5 doesn't exist, so we can just use that name.

Not sure if you intentionally kept the suffix, I'd prefer removing it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scripts that add licenses add this suffix. I think the suffix has advantage of clearly marking the image as different from the original Argo image.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM, if you find the extra information helpful

id: 'pullArgoExecutor'
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance']
id: 'pullArgoWorkflowController'

# Tag for Hosted - SemVersion to Major.Minor parsing
Expand Down
20 changes: 10 additions & 10 deletions .release.cloudbuild.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,14 +438,14 @@ steps:
docker push gcr.io/ml-pipeline/google/pipelines-test/cloudsqlproxy:$(cat /workspace/mm.ver)

- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance']
id: 'pullArgoExecutor'
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoexecutor:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoexecutor:$TAG_NAME']
id: 'tagArgoExecutorForMarketplace'
waitFor: ['pullArgoExecutor']
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$TAG_NAME']
id: 'tagArgoExecutorForMarketplaceTest'
waitFor: ['pullArgoExecutor']
- id: 'tagArgoExecutorForMarketplaceMajorMinor'
Expand All @@ -455,20 +455,20 @@ steps:
args:
- -ceux
- |
docker tag gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance gcr.io/ml-pipeline/google/pipelines/argoexecutor:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance gcr.io/ml-pipeline/google/pipelines/argoexecutor:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines/argoexecutor:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$(cat /workspace/mm.ver)

- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance']
id: 'pullArgoWorkflowController'
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$TAG_NAME']
id: 'tagArgoWorkflowControllerForMarketplace'
waitFor: ['pullArgoWorkflowController']
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$TAG_NAME']
id: 'tagArgoWorkflowControllerForMarketplaceTest'
waitFor: ['pullArgoWorkflowController']
- id: 'tagArgoWorkflowControllerForMarketplaceMajorMinor'
Expand All @@ -478,8 +478,8 @@ steps:
args:
- -ceux
- |
docker tag gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$(cat /workspace/mm.ver)

Expand Down
3 changes: 2 additions & 1 deletion backend/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ COPY backend/requirements.txt .
RUN python3 -m pip install -r requirements.txt

# Downloading Argo CLI so that the samples are validated
ADD https://github.com/argoproj/argo/releases/download/v2.3.0/argo-linux-amd64 /usr/local/bin/argo
#ADD https://github.com/argoproj/argo/releases/download/v2.7.5/argo-linux-amd64 /usr/local/bin/argo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment here, why we need to use the old argo CLI?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sure. I'll do that.
The reason is that the Argo CLI is only used for validation (argo lint) and later versions of Argo CLI can no longer work inside a container.

ADD https://github.com/argoproj/argo/releases/download/v2.4.3/argo-linux-amd64 /usr/local/bin/argo
RUN chmod +x /usr/local/bin/argo

WORKDIR /go/src/github.com/kubeflow/pipelines
Expand Down
53 changes: 53 additions & 0 deletions manifests/gcp_marketplace/chart/kubeflow-pipelines/templates/argo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,56 @@ spec:
scope: Namespaced
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: clusterworkflowtemplates.argoproj.io
labels:
app.kubernetes.io/name: {{ .Release.Name }}
spec:
group: argoproj.io
names:
kind: ClusterWorkflowTemplate
plural: clusterworkflowtemplates
shortNames:
- clusterwftmpl
- cwft
scope: Cluster
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cronworkflows.argoproj.io
labels:
app.kubernetes.io/name: {{ .Release.Name }}
spec:
group: argoproj.io
names:
kind: CronWorkflow
plural: cronworkflows
shortNames:
- cronwf
- cwf
scope: Namespaced
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: workflowtemplates.argoproj.io
labels:
app.kubernetes.io/name: {{ .Release.Name }}
spec:
group: argoproj.io
names:
kind: WorkflowTemplate
plural: workflowtemplates
shortNames:
- wftmpl
scope: Namespaced
version: v1alpha1
---
apiVersion: v1
kind: ServiceAccount
metadata:
Expand Down Expand Up @@ -61,6 +111,9 @@ rules:
resources:
- workflows
- workflows/finalizers
- workflowtemplates
- workflowtemplates/finalizers
- cronworkflows
verbs:
- get
- list
Expand Down
56 changes: 55 additions & 1 deletion manifests/kustomize/base/argo/cluster-scoped/workflow-crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,58 @@ spec:
kind: Workflow
plural: workflows
shortNames:
- wf
- wf
additionalPrinterColumns:
- JSONPath: .status.phase
description: Status of the workflow
name: Status
type: string
- JSONPath: .status.startedAt
description: When the workflow was started
format: date-time
name: Age
type: date
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: clusterworkflowtemplates.argoproj.io
spec:
group: argoproj.io
names:
kind: ClusterWorkflowTemplate
plural: clusterworkflowtemplates
shortNames:
- clusterwftmpl
- cwft
scope: Cluster
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cronworkflows.argoproj.io
spec:
group: argoproj.io
names:
kind: CronWorkflow
plural: cronworkflows
shortNames:
- cronwf
- cwf
scope: Namespaced
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: workflowtemplates.argoproj.io
spec:
group: argoproj.io
names:
kind: WorkflowTemplate
plural: workflowtemplates
shortNames:
- wftmpl
scope: Namespaced
version: v1alpha1
2 changes: 1 addition & 1 deletion manifests/kustomize/base/argo/workflow-controller-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ data:
config: |
{
namespace: $(NAMESPACE),
executorImage: gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance,
executorImage: gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance,
artifactRepository:
{
s3: {
Expand Down
4 changes: 2 additions & 2 deletions manifests/kustomize/base/argo/workflow-controller-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ spec:
- --configmap
- workflow-controller-configmap
- --executor-image
- gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance
- gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance
command:
- workflow-controller
env:
Expand All @@ -36,7 +36,7 @@ spec:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance
image: gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance
imagePullPolicy: IfNotPresent
name: workflow-controller
resources: {}
Expand Down
3 changes: 3 additions & 0 deletions manifests/kustomize/base/argo/workflow-controller-role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ rules:
resources:
- workflows
- workflows/finalizers
- workflowtemplates
- workflowtemplates/finalizers
- cronworkflows
verbs:
- get
- list
Expand Down
2 changes: 1 addition & 1 deletion test/check-argo-status.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ if [[ "$workflow_completed" == "false" ]] || [[ "$workflow_failed" != "false" ]]
fi

echo "=========Argo Workflow Logs========="
argo logs -w "${ARGO_WORKFLOW}" -n "${NAMESPACE}"
argo logs "${ARGO_WORKFLOW}" -n "${NAMESPACE}"

echo "========All workflows============="

Expand Down
2 changes: 1 addition & 1 deletion test/install-argo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ACCOUNT=$(gcloud info --format='value(config.account)')
kubectl create clusterrolebinding PROW_BINDING --clusterrole=cluster-admin --user=$ACCOUNT --dry-run -o yaml | kubectl apply -f -
kubectl create clusterrolebinding DEFAULT_BINDING --clusterrole=cluster-admin --serviceaccount=default:default --dry-run -o yaml | kubectl apply -f -

ARGO_VERSION=v2.3.0
ARGO_VERSION=v2.7.5

# if argo is not installed
if ! which argo; then
Expand Down
3 changes: 2 additions & 1 deletion test/sample-test/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ RUN cd /python/src/github.com/kubeflow/pipelines
# Install argo.
RUN echo "install argo"
RUN mkdir -p /usr/local/bin/
RUN ARGO_VERSION=v2.3.0 && curl -sSL -o /usr/local/bin/argo \
#RUN ARGO_VERSION=v2.7.5 && curl -sSL -o /usr/local/bin/argo \
RUN ARGO_VERSION=v2.4.3 && curl -sSL -o /usr/local/bin/argo \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a comment here too?

"https://github.com/argoproj/argo/releases/download/$ARGO_VERSION/argo-linux-amd64" && \
chmod +x /usr/local/bin/argo
ENV PATH $PATH:/usr/local/bin/argo
Expand Down
8 changes: 4 additions & 4 deletions test/tag_for_hosted.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,12 +112,12 @@ docker tag gcr.io/cloudsql-docker/gce-proxy:1.14 gcr.io/$PROJECT_ID/hosted/$COMM
docker push gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/cloudsqlproxy:$SEM_VER
docker push gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/cloudsqlproxy:$MM_VER

docker tag gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoexecutor:$SEM_VER
docker tag gcr.io/ml-pipeline/argoexec:v2.3.0-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoexecutor:$MM_VER
docker tag gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoexecutor:$SEM_VER
docker tag gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoexecutor:$MM_VER
docker push gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoexecutor:$SEM_VER
docker push gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoexecutor:$MM_VER

docker tag gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoworkflowcontroller:$SEM_VER
docker tag gcr.io/ml-pipeline/workflow-controller:v2.3.0-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoworkflowcontroller:$MM_VER
docker tag gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoworkflowcontroller:$SEM_VER
docker tag gcr.io/ml-pipeline/workflow-controller:v2.7.5-license-compliance gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoworkflowcontroller:$MM_VER
docker push gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoworkflowcontroller:$SEM_VER
docker push gcr.io/$PROJECT_ID/hosted/$COMMIT_SHA/argoworkflowcontroller:$MM_VER
2 changes: 1 addition & 1 deletion third_party/argo/Dockerfile.argoexec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ RUN sh /third_party/download_source.sh </third_party/argo/repo-MPL.txt


# Argo exec image
From argoproj/argoexec:v2.3.0
From argoproj/argoexec:v2.7.5

# Copy concatenated license file
COPY third_party/argo/license.txt /third_party/license.txt
Expand Down
2 changes: 1 addition & 1 deletion third_party/argo/Dockerfile.workflow-controller
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ RUN apk add --no-cache curl
ENV OUTPUT_FOLDER /downloads/
RUN sh /third_party/download_source.sh </third_party/argo/repo-MPL.txt

From argoproj/workflow-controller:v2.3.0
From argoproj/workflow-controller:v2.7.5

# Copy concatenated license file
COPY third_party/argo/license.txt /third_party/license.txt
Expand Down
2 changes: 1 addition & 1 deletion third_party/argo/release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ set -e
# $PROJECT: gcp project

RELEASE_PROJECT=ml-pipeline
TAG=v2.3.0-license-compliance
TAG=v2.7.5-license-compliance

gcloud builds submit --config third_party/argo/cloudbuild.yaml . \
--substitutions=TAG_NAME="$TAG" --project "$PROJECT"
Expand Down