Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Cache Deployer] crashloopbackoff in Kubernetes 1.17 #4627

Closed
Bobgy opened this issue Oct 14, 2020 · 4 comments · Fixed by #4632
Closed

[Cache Deployer] crashloopbackoff in Kubernetes 1.17 #4627

Bobgy opened this issue Oct 14, 2020 · 4 comments · Fixed by #4632
Assignees
@Ark-kun
Labels
area/execution_cache kind/bug priority/p0

Comments

@Bobgy
Copy link
Contributor

Bobgy commented Oct 14, 2020 •
edited
Loading

This is blocking KFP 1.0.3 release on GCP marketplace.
#4540

What steps did you take:

[A clear and concise description of what the bug is.]

  1. Go to https://console.cloud.google.com/marketplace/details/google-cloud-ai-platform/kubeflow-pipelines (in fact, the test solution that's on 1.0.3)
  2. Create cluster there (it will use regular channel and get 1.17)
  3. Deploy KFP

What happened:

cache deployer keeps crashing

  • kubectl apply -f ./cache-configmap-ca-bundle.yaml --namespace default
    error: error validating "./cache-configmap-ca-bundle.yaml": error validating data: [ValidationError(MutatingWebhookConfiguration.webhooks[0]): missing required field "sideEffects" in io.k8s.api.admissionregistration.v1.MutatingWebhook, ValidationError(MutatingWebhookConfiguration.webhooks[0]): missing required field "admissionReviewVersions" in io.k8s.api.admissionregistration.v1.MutatingWebhook]; if you choose to ignore these errors, turn validation off with --validate=false
    Error

What did you expect to happen:

cache deployer should succeed

Environment:

How did you deploy Kubeflow Pipelines (KFP)?

AI Platform Pipelines test solution with 1.0.3

KFP version:
1.0.3

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug
@Bobgy
Copy link
Contributor Author

Bobgy commented Oct 14, 2020

/assign @Ark-kun

@Bobgy Bobgy mentioned this issue Oct 14, 2020
Closed
3 tasks
@Bobgy
Copy link
Contributor Author

Bobgy commented Oct 14, 2020

I'm updating presubmit tests to regular channel to showcase the problem: #4626

@Ark-kun Ark-kun mentioned this issue Oct 17, 2020
Merged
@Ark-kun
Copy link
Contributor

Ark-kun commented Oct 17, 2020 •
edited
Loading

2. Create cluster there (it will use regular channel and get 1.17)

This is a bit strange. When I try to create a cluster it uses the stable version 1.16 by default.

image

Update:
But if I switch to "Release channel", then it seems to default to Regular which is 1.17.

Maybe there was some recent change.

@Bobgy
Copy link
Contributor Author

Bobgy commented Oct 19, 2020 •
edited
Loading

@Ark-kun I think the default version for creating a cluster from MKP is different from GKE UI.
MKP auto creates regular channel from my experiments.

@Bobgy Bobgy mentioned this issue Oct 19, 2020
Closed
k8s-ci-robot pushed a commit that referenced this issue Oct 20, 2020
@Ark-kun
fix(backend): Caching - Fixed deployer failure on Kubernetes v1.16+. F…
dde7f9a 
…ixes #4627 (#4632)

* Backend - Caching - Fixed deployer failure on Kubernetes v1.16+

The sideEffects field field became required in v1 version of the resource kubernetes/kubernetes#79549

Also adding failurePolicy: Ignore, because the default value has changed to Fail in v1.16.

These changes are not needed for v1beta1, but I still add them for those cases as well for consistency.

* The admissionReviewVersions field became required in the v1 API in v1.16

See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request
Bobgy pushed a commit to Bobgy/pipelines that referenced this issue Oct 22, 2020
@Ark-kun @Bobgy
fix(backend): Caching - Fixed deployer failure on Kubernetes v1.16+. F…
a599d32 
…ixes kubeflow#4627 (kubeflow#4632)

* Backend - Caching - Fixed deployer failure on Kubernetes v1.16+

The sideEffects field field became required in v1 version of the resource kubernetes/kubernetes#79549

Also adding failurePolicy: Ignore, because the default value has changed to Fail in v1.16.

These changes are not needed for v1beta1, but I still add them for those cases as well for consistency.

* The admissionReviewVersions field became required in the v1 API in v1.16

See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request
Bobgy pushed a commit to Bobgy/pipelines that referenced this issue Oct 22, 2020
@Ark-kun @Bobgy
fix(backend): Caching - Fixed deployer failure on Kubernetes v1.16+. F…
c21587b 
…ixes kubeflow#4627 (kubeflow#4632)

* Backend - Caching - Fixed deployer failure on Kubernetes v1.16+

The sideEffects field field became required in v1 version of the resource kubernetes/kubernetes#79549

Also adding failurePolicy: Ignore, because the default value has changed to Fail in v1.16.

These changes are not needed for v1beta1, but I still add them for those cases as well for consistency.

* The admissionReviewVersions field became required in the v1 API in v1.16

See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request
Bobgy pushed a commit that referenced this issue Oct 22, 2020
@Ark-kun @Bobgy
fix(backend): Caching - Fixed deployer failure on Kubernetes v1.16+. F…
1792103 
…ixes #4627 (#4632)

* Backend - Caching - Fixed deployer failure on Kubernetes v1.16+

The sideEffects field field became required in v1 version of the resource kubernetes/kubernetes#79549

Also adding failurePolicy: Ignore, because the default value has changed to Fail in v1.16.

These changes are not needed for v1beta1, but I still add them for those cases as well for consistency.

* The admissionReviewVersions field became required in the v1 API in v1.16

See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request
Jeffwan pushed a commit to Jeffwan/pipelines that referenced this issue Dec 9, 2020
@Ark-kun @Jeffwan
fix(backend): Caching - Fixed deployer failure on Kubernetes v1.16+. F…
e841f0e 
…ixes kubeflow#4627 (kubeflow#4632)

* Backend - Caching - Fixed deployer failure on Kubernetes v1.16+

The sideEffects field field became required in v1 version of the resource kubernetes/kubernetes#79549

Also adding failurePolicy: Ignore, because the default value has changed to Fail in v1.16.

These changes are not needed for v1beta1, but I still add them for those cases as well for consistency.

* The admissionReviewVersions field became required in the v1 API in v1.16

See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ark-kun Ark-kun

Labels
area/execution_cache kind/bug priority/p0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(backend): Caching - Fixed deployer failure on Kubernetes v1.16+. Fixes #4627 Ark-kun/pipelines
3 participants
@Ark-kun @Bobgy @k8s-ci-robot