Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend - Marking auto-added artifacts as optional #1289

Conversation

Ark-kun
Copy link
Contributor

@Ark-kun Ark-kun commented May 7, 2019

This PR is implementing part of the #1250 plan.

Problem:
DSL-compiler has been auto-adding two artifacts to all pipeline tasks (including ones that do not actually outputs those artifacts). All out existing compiled pipelines have this problem.
Argo 2.3 makes missing output artifacts a reason for task failure.
The workaround for the existing pipelines is for backend to mark those two artifacts as optional.

P.S.
The optional property is only available starting with Argo 2.3, so I'm bumping the Argo version.


This change is Reviewable

@Ark-kun
Copy link
Contributor Author

Ark-kun commented May 7, 2019

The tests are currently failing since they use the wrong Argo version (despite updated go.mod). @neuromage What do I need to change to make Bazel use the proper Argo version?

@neuromage
Copy link
Contributor

You can run

bazel run //:gazelle -- --update-repos --from_file=go.mod

@Ark-kun
Copy link
Contributor Author

Ark-kun commented May 7, 2019

You can run

bazel run //:gazelle -- --update-repos --from_file=go.mod

Thanks.
I did that yesterday, but bazel fails for me:

~/go/src/github.com/kubeflow/pipelines$ bazel run //:gazelle -- --update-repos --from_file=go.mod
Starting local Bazel server and connecting to it...
ERROR: /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/31e08d5c4be525ad16464e998010eee7/external/org_tensorflow/third_party/gpus/cuda_configure.bzl:115:1: load() statements must be called before any other statement. Firs
t non-load() statement appears at /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/31e08d5c4be525ad16464e998010eee7/external/org_tensorflow/third_party/gpus/cuda_configure.bzl:26:1. Use --incompatible_bzl_disallow_load_af
ter_statement=false to temporarily disable this check.
ERROR: error loading package '': in /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/31e08d5c4be525ad16464e998010eee7/external/org_tensorflow/tensorflow/workspace.bzl: Extension 'third_party/gpus/cuda_configure.bzl' has e
rrors
ERROR: error loading package '': in /usr/local/google/home/avolkov/.cache/bazel/_bazel_avolkov/31e08d5c4be525ad16464e998010eee7/external/org_tensorflow/tensorflow/workspace.bzl: Extension 'third_party/gpus/cuda_configure.bzl' has e
rrors
INFO: Elapsed time: 2.586s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

@Ark-kun
Copy link
Contributor Author

Ark-kun commented May 7, 2019

TF has fixed the issue ~1 month ago tensorflow/tensorflow@494d2cc It's part of the v1.12.2 release, but not v1.13.1.

@k8s-ci-robot k8s-ci-robot added size/M and removed size/S labels May 7, 2019
@Ark-kun
Copy link
Contributor Author

Ark-kun commented May 8, 2019

All the tests are passing.

@Ark-kun Ark-kun changed the title [WIP]Backend - Marking auto-added artifacts as optional Backend - Marking auto-added artifacts as optional May 8, 2019
WORKSPACE Outdated Show resolved Hide resolved
@neuromage
Copy link
Contributor

/lgtm

go.mod Outdated Show resolved Hide resolved
Copy link
Member

@IronPan IronPan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

backend/src/apiserver/resource/resource_manager.go Outdated Show resolved Hide resolved
@IronPan
Copy link
Member

IronPan commented May 9, 2019

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit b2d9a98 into kubeflow:master May 9, 2019
@Ark-kun Ark-kun deleted the Backend---Marking-auto-added-artifacts-as-optional branch May 9, 2019 22:07
@Ark-kun
Copy link
Contributor Author

Ark-kun commented May 9, 2019

@IronPan Looks like there is a problem in how Kubeflow deploys Argo. They deploy two workflow controllers with different configurations (one in argo namespace and another one in kubeflow namespaces). KF Pipelines seems to be using the one from argo namespace which is not configured using configMap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants