Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Package server APIServer unreachable after install on EKS with Istio #2514

Open
amilanoski opened this issue Dec 8, 2021 · 2 comments
Open

Package server APIServer unreachable after install on EKS with Istio #2514

amilanoski opened this issue Dec 8, 2021 · 2 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. triaged Issue has been considered by a member of the OLM community

Comments

@amilanoski
Copy link

amilanoski commented Dec 8, 2021 •
edited
Loading

Bug Report

What did you do?
A clear and concise description of the steps you took (or insert a code snippet).

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.19.1/crds.yaml
  - https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.19.1/olm.yaml

kubectl apply -k .

What did you expect to see?
A clear and concise description of what you expected to happen (or insert a code snippet).

OLM Services come up and I am able to install an Operator from OperatorHub.io

What did you see instead? Under which circumstances?
A clear and concise description of what you expected to happen (or insert a code snippet).

OLM Services came up according to this.

kg pod,svc -n olm
NAME                                    READY   STATUS    RESTARTS   AGE
pod/catalog-operator-765c45774c-dsvm9   2/2     Running   1          166m
pod/olm-operator-9bb87877b-zmr7f        2/2     Running   1          166m
pod/operatorhubio-catalog-fq7qn         2/2     Running   0          61m
pod/packageserver-568d867fff-bzsjb      2/2     Running   0          13m
pod/packageserver-568d867fff-t7hs8      2/2     Running   0          62m

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/operatorhubio-catalog   ClusterIP   10.100.177.142   <none>        50051/TCP   166m
service/packageserver-service   ClusterIP   10.100.127.51    <none>        5443/TCP    2m22s

packserver csv stuck in Installing State

kg csv -n olm
NAME            DISPLAY          VERSION   REPLACES   PHASE
packageserver   Package Server   0.19.1               Installing

  Requirement Status:
    Group:    operators.coreos.com
    Kind:     ClusterServiceVersion
    Message:  CSV minKubeVersion (1.11.0) less than server version (v1.20.7-eks-d88609)
    Name:     packageserver
    Status:   Present
    Version:  v1alpha1
    Group:    apiregistration.k8s.io
    Kind:     APIService
    Message:
    Name:     v1.packages.operators.coreos.com
    Status:   DeploymentFound
    Version:  v1
    Dependents:
      Group:    rbac.authorization.k8s.io
      Kind:     PolicyRule
      Message:  cluster rule:{"verbs":["create","get"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
      Status:   Satisfied
      Version:  v1
      Group:    rbac.authorization.k8s.io
      Kind:     PolicyRule
      Message:  cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["configmaps"]}
      Status:   Satisfied
      Version:  v1
      Group:    rbac.authorization.k8s.io
      Kind:     PolicyRule
      Message:  cluster rule:{"verbs":["get","list","watch"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}
      Status:   Satisfied
      Version:  v1
      Group:    rbac.authorization.k8s.io
      Kind:     PolicyRule
      Message:  cluster rule:{"verbs":["get","list"],"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"]}
      Status:   Satisfied
      Version:  v1
    Group:
    Kind:       ServiceAccount
    Message:
    Name:       olm-operator-serviceaccount
    Status:     Present
    Version:    v1
Events:
  Type     Reason               Age                   From                        Message
  ----     ------               ----                  ----                        -------
  Normal   RequirementsUnknown  52m                   operator-lifecycle-manager  requirements not yet checked
  Normal   AllRequirementsMet   42m (x6 over 52m)     operator-lifecycle-manager  all requirements found, attempting install
  Normal   NeedsReinstall       42m (x4 over 47m)     operator-lifecycle-manager  APIServices not installed
  Normal   InstallSucceeded     42m (x6 over 52m)     operator-lifecycle-manager  waiting for install components to report healthy
  Normal   InstallWaiting       42m (x5 over 52m)     operator-lifecycle-manager  APIServices not installed
  Warning  InstallCheckFailed   2m17s (x19 over 47m)  operator-lifecycle-manager  install timeout

kl -n olm olm-operator-9bb87877b-zmr7f --tail=50 -f

time="2021-12-08T20:17:00Z" level=info msg="install strategy successful" csv=packageserver id=bVi6s namespace=olm phase=Installing strategy=deployment
time="2021-12-08T20:17:00Z" level=info msg="install strategy successful" csv=packageserver id=Su0tW namespace=olm phase=Installing strategy=deployment
time="2021-12-08T20:17:01Z" level=warning msg="install timed out" csv=packageserver id=AzvRi namespace=olm phase=Installing
I1208 20:17:01.095196       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147367595", FieldPath:""}): type: 'Warning' reason: 'InstallCheckFailed' install timeout
time="2021-12-08T20:17:01Z" level=warning msg="install timed out" csv=packageserver id=eYZNN namespace=olm phase=Installing
I1208 20:17:01.250682       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147367595", FieldPath:""}): type: 'Warning' reason: 'InstallCheckFailed' install timeout
time="2021-12-08T20:17:01Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=SldDi namespace=olm phase=Installing
E1208 20:17:01.267190       1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:01Z" level=warning msg="needs reinstall: APIServices not installed" csv=packageserver id=kLQmP namespace=olm phase=Failed strategy=deployment
I1208 20:17:01.445074       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369389", FieldPath:""}): type: 'Normal' reason: 'NeedsReinstall' APIServices not installed
time="2021-12-08T20:17:01Z" level=warning msg="needs reinstall: APIServices not installed" csv=packageserver id=owVzw namespace=olm phase=Failed strategy=deployment
I1208 20:17:01.646871       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369389", FieldPath:""}): type: 'Normal' reason: 'NeedsReinstall' APIServices not installed
time="2021-12-08T20:17:01Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=ePxEX namespace=olm phase=Failed
E1208 20:17:01.655655       1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:01Z" level=info msg="scheduling ClusterServiceVersion for install" csv=packageserver id=4rhbL namespace=olm phase=Pending
I1208 20:17:01.748610       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369396", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install
time="2021-12-08T20:17:01Z" level=info msg="scheduling ClusterServiceVersion for install" csv=packageserver id=h2fyb namespace=olm phase=Pending
I1208 20:17:01.844415       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369396", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install
time="2021-12-08T20:17:01Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=q0q+v namespace=olm phase=Pending
E1208 20:17:01.852175       1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:01Z" level=warning msg="reusing existing cert packageserver-service-cert"
I1208 20:17:02.010812       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369400", FieldPath:""}): type: 'Normal' reason: 'InstallSucceeded' waiting for install components to report healthy
time="2021-12-08T20:17:02Z" level=warning msg="reusing existing cert packageserver-service-cert"
I1208 20:17:02.191973       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369400", FieldPath:""}): type: 'Normal' reason: 'InstallSucceeded' waiting for install components to report healthy
time="2021-12-08T20:17:02Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=FSdCV namespace=olm phase=InstallReady
E1208 20:17:02.202441       1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:02Z" level=info msg="install strategy successful" csv=packageserver id=UY7It namespace=olm phase=Installing strategy=deployment
I1208 20:17:02.221591       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369412", FieldPath:""}): type: 'Normal' reason: 'InstallWaiting' APIServices not installed
time="2021-12-08T20:17:02Z" level=info msg="install strategy successful" csv=packageserver id=PqBQQ namespace=olm phase=Installing strategy=deployment
I1208 20:17:02.248508       1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369412", FieldPath:""}): type: 'Normal' reason: 'InstallWaiting' APIServices not installed
time="2021-12-08T20:17:02Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=kebtg namespace=olm phase=Installing
E1208 20:17:02.258585       1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:02Z" level=info msg="install strategy successful" csv=packageserver id=x5s4G namespace=olm phase=Installing strategy=deployment

Environment

  • operator-lifecycle-manager version: 0.19.1
  • Kubernetes version information: 1.20 EKS

Istio 1.11.3

kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.7-eks-d88609", GitCommit:"d886092805d5cc3a47ed5cf0c43de38ce442dfcb", GitTreeState:"clean", BuildDate:"2021-07-31T00:29:12Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes cluster kind:

Possible Solution

Additional context
Add any other context about the problem here.

Possibly Related
#1368
#2343
#2234
@amilanoski amilanoski added the kind/bug Categorizes issue or PR as related to a bug. label Dec 8, 2021
@dinhxuanvu
Copy link
Member

Hi there,
Unfortunately, there is not enough information on the description to identify what the root cause is. I would like to ask for the pod/deployment yaml blob for the packageserver on the olm namespace.

@dinhxuanvu dinhxuanvu added the triaged Issue has been considered by a member of the OLM community label Dec 9, 2021
@martin31821
Copy link

Hey, had a similar issue
using cilium, pushed a fix into my fork. Problem essentially is that EKS can't connect to the packageserver API, which I've solved by using hostNetwork on these pods.: master...deinstapel:operator-lifecycle-manager:v0.23.1-fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. triaged Issue has been considered by a member of the OLM community
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@martin31821 @dinhxuanvu @amilanoski