Che fails to deploy on minikube v1.19.0 #19602

sparkoo · 2021-04-15T07:22:29Z

Describe the bug

On minikube v1.19.0, Che fails to deploy. Chectl gets stuck at scheduling keycloak and che-operator log has some errors with ingresses. See info below.

Workaround: Minikube v1.18.1 works fine.

Che version

nightly

Steps to reproduce

Install and start minikube v1.19.0
run chectl server:deploy --platform=minikube
deployment get stuck on scheduling keycloak

Expected behavior

deploy works

Runtime

minikube (include output of minikube version and kubectl version)\

[~] λ minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5

Screenshots

Installation method

chectl

chectl server:deploy --platform=minikube

Environment

my computer
- Linux

Eclipse Che Logs

chectl stuck at:

    ❯ Keycloak pod bootstrap
      ⠹ Scheduling
        Downloading images
        Starting

che-operator:

time="2021-04-15T07:20:45Z" level=info msg="Creating a new object: Service, name: che-host"
time="2021-04-15T07:20:45Z" level=info msg="Creating a new object: Ingress, name: che"
time="2021-04-15T07:20:45Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-15T07:20:45Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-15T07:20:47Z" level=info msg="Creating a new object: Ingress, name: che"
time="2021-04-15T07:20:47Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-15T07:20:47Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-15T07:20:49Z" level=info msg="Creating a new object: Ingress, name: che"
time="2021-04-15T07:20:49Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-15T07:20:49Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-15T07:20:51Z" level=info msg="Creating a new object: Ingress, name: che"
time="2021-04-15T07:20:51Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-15T07:20:51Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"

Additional context

The text was updated successfully, but these errors were encountered:

nickboldt · 2021-04-19T12:22:56Z

@flacatus is this the reason why chectl HP tests tend to fail? If so, should we...

disable the PR check test that doens't pass consistently?
rewrite it so it uses older minikube version (and is therefore stable) ?
fix the PR check tests so they work w/ latest minikube 1.19?

Also what's the timeline for this ? it would be good to set a milestone / sprint # so we know if/when this is going to be worked on.

flacatus · 2021-04-28T08:13:07Z

Hi @nickboldt and @sparkoo !
Sorry for delay response...

Yeah, we are installing in our tests for now minikube 1.18.1. See https://github.com/eclipse-che/che-operator/blob/main/.github/workflows/minikube-operator-update.yaml#L28-L29

Digging a little bit more in the issue I've discovered in minikube 1.19.0 exists a bug with ingress. More info about the bug can be find in this minikube issue: kubernetes/minikube#11147 .

Also I see in minikube repo already was provided a fix for ingress addon and in new minikube version ¿1.19.1/1.20.0? the ingress problem should be fixed: kubernetes/minikube#11189.

flacatus · 2021-04-28T08:18:15Z

Logs from nginx controller:

I0428 07:20:30.050068       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5d88495688-tb4f4", UID:"8e08f375-fea0-43cc-a425-7def05059bef", APIVersion:"v1", ResourceVersion:"636", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
E0428 07:21:48.061419       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:21:49.674249       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:21:51.605296       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:21:53.604383       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:21:55.589893       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:21:57.616696       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:21:59.629902       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:22:01.615750       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
E0428 07:22:03.642739       7 server.go:84] "failed to process webhook request" err="rejecting admission review because the request does not contain an Ingress resource but networking.k8s.io/v1, Kind=Ingress with name che in namespace eclipse-che"
.
.
.

And che-operator logs:

time="2021-04-28T07:21:48Z" level=info msg="Creating a new object: v1beta1.Ingress, name: che"
time="2021-04-28T07:21:48Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-28T07:21:48Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-28T07:21:49Z" level=info msg="Creating a new object: v1beta1.Ingress, name: che"
time="2021-04-28T07:21:49Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-28T07:21:49Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-28T07:21:51Z" level=info msg="Creating a new object: v1beta1.Ingress, name: che"
time="2021-04-28T07:21:51Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-28T07:21:51Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-28T07:21:53Z" level=info msg="Creating a new object: v1beta1.Ingress, name: che"
time="2021-04-28T07:21:53Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-28T07:21:53Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-28T07:21:55Z" level=info msg="Creating a new object: v1beta1.Ingress, name: che"
time="2021-04-28T07:21:55Z" level=info msg="Waiting on ingress 'che' to be ready"
time="2021-04-28T07:21:55Z" level=error msg="Internal error occurred: failed calling webhook \"validate.nginx.ingress.kubernetes.io\": an error on the server (\"\") has prevented the request from succeeding"
time="2021-04-28T07:21:57Z" level=info msg="Creating a new object: v1beta1.Ingress, name: che"
time="2021-04-28T07:21:57Z" level=info msg="Waiting on ingress 'che' to be ready"
.
.
.

sparkoo · 2021-05-07T10:23:54Z

seems working on minikube 1.20.0

flacatus · 2021-05-10T10:04:41Z

@sparkoo thanks for reporting.
I can see in 1.20.0 is working properly. I will close the issue.

› Current Kubernetes context: 'minikube'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Check Kubernetes version: Found v1.20.2.
    ✔ Verify if minikube ingress addon is enabled
    ↓ Enable minikube ingress addon [skipped]
      → Ingress addon is already enabled.
    ✔ Retrieving minikube IP and domain for ingress URLs...192.168.39.193.nip.io.
    ✔ Checking minikube version... 1.20.0
    ↓ Check if cluster accessible [skipped]
  ✔ Following Eclipse Che logs
    ✔ Start following Operator logs...done
    ✔ Start following Eclipse Che Server logs...done
    ✔ Start following PostgreSQL logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plug-in Registry logs...done
    ✔ Start following Devfile Registry logs...done
    ✔ Start following Eclipse Che Dashboard logs...done
    ✔ Start following namespace events...done
  ✔ 🏃‍  Running the Eclipse Che operator
    ✔ Create Namespace (eclipse-che)...Done.
    ✔ Create ServiceAccount che-operator in namespace eclipse-che...done.
    ✔ Read Roles and Bindings...done.
    ✔ Creating Roles and Bindings...done.
    ✔ Create CRD checlusters.org.eclipse.che...done.
    ✔ Create CRD chemanagers.che.eclipse.org...Skipped.
    ✔ Create CRD devworkspaceroutings.controller.devfile.io...Skipped.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    ✔ Create deployment che-operator in namespace eclipse-che...done.
    ✔ Operator pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Prepare Eclipse Che cluster CR...Done.
    ✔ Create the Custom Resource of type checlusters.org.eclipse.che in the namespace eclipse-che...done.
  ✔ ✅  Post installation checklist
    ✔ PostgreSQL pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Keycloak pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Devfile Registry pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Plug-in Registry pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Eclipse Che Dashboard pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Eclipse Che Server pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Eclipse Che status check...done
  ✔ Retrieving Keycloak admin credentials...done
  ✔ Retrieving Che self-signed CA certificate... done
  ✔ Prepare post installation output...done
  ✔ Show important messages
    ✔ Eclipse Che 7.30.0-SNAPSHOT has been successfully deployed.
    ✔ Documentation             : https://www.eclipse.org/che/docs/
    ✔ -------------------------------------------------------------------------------
    ✔ Users Dashboard           : https://che-eclipse-che.192.168.39.193.nip.io
    ✔ Admin user login          : "admin:admin". NOTE: must change after first login.
    ✔ -------------------------------------------------------------------------------
    ✔ Plug-in Registry          : https://plugin-registry-eclipse-che.192.168.39.193.nip.io/v3
    ✔ Devfile Registry          : https://devfile-registry-eclipse-che.192.168.39.193.nip.io/
    ✔ -------------------------------------------------------------------------------
    ✔ Identity Provider URL     : https://keycloak-eclipse-che.192.168.39.193.nip.io/auth
    ✔ Identity Provider login   : "admin:fsQiYEWZ1X5r".
    ✔ -------------------------------------------------------------------------------
    ✔ [ACTION REQUIRED] Please add Che self-signed CA certificate into your browser: /tmp/cheCA.crt.
   Documentation how to add a CA certificate into a browser: https://www.eclipse.org/che/docs/che-7/end-user-guide/importing-certificates-to-browsers/
Command server:deploy has completed successfully in 04:47.

sparkoo added kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system. area/chectl Issues related to chectl, the CLI of Che area/che-operator Issues and PRs related to Eclipse Che Kubernetes Operator labels Apr 15, 2021

tolusha added the sprint/next label Apr 15, 2021

This was referenced Apr 15, 2021

Install minikube 1.18.1 instead of latest one eclipse-che/che-operator#780

Merged

fix: Install minikube 1.18.1 instead of latest one che-incubator/chectl#1180

Merged

This was referenced Apr 15, 2021

chore(happy-path): stick minikube to a working version eclipse-che/che-theia#1080

Merged

chore(happy-path): stick minikube to a working version eclipse-che/che-plugin-registry#932

Merged

flacatus mentioned this issue Apr 19, 2021

Release version 7.29.0 che-incubator/chectl#1182

Merged

flacatus mentioned this issue Apr 26, 2021

Release version 7.29.1 che-incubator/chectl#1196

Merged

flacatus self-assigned this Apr 28, 2021

tolusha removed the sprint/next label Apr 29, 2021

flacatus closed this as completed May 10, 2021

sympatheticmoose added the new&noteworthy For new and/or noteworthy issues that deserve a blog post, new docs, or emphasis in release notes label May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Che fails to deploy on minikube v1.19.0 #19602

Che fails to deploy on minikube v1.19.0 #19602

sparkoo commented Apr 15, 2021 •

edited

Loading

nickboldt commented Apr 19, 2021

flacatus commented Apr 28, 2021 •

edited

Loading

flacatus commented Apr 28, 2021

sparkoo commented May 7, 2021

flacatus commented May 10, 2021

Che fails to deploy on minikube v1.19.0 #19602

Che fails to deploy on minikube v1.19.0 #19602

Comments

sparkoo commented Apr 15, 2021 • edited Loading

Describe the bug

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

Eclipse Che Logs

chectl stuck at:

che-operator:

Additional context

nickboldt commented Apr 19, 2021

flacatus commented Apr 28, 2021 • edited Loading

flacatus commented Apr 28, 2021

sparkoo commented May 7, 2021

flacatus commented May 10, 2021

sparkoo commented Apr 15, 2021 •

edited

Loading

flacatus commented Apr 28, 2021 •

edited

Loading