Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed install pixie #668

Open
zainal-abidin-assegaf opened this issue Dec 15, 2022 · 6 comments
Open

failed install pixie #668

zainal-abidin-assegaf opened this issue Dec 15, 2022 · 6 comments
Labels
area/deployment Issues replated to deployments kind/support Categorizes issue or PR as a support question. triage/accepted Indicates an issue or PR is ready to be actively worked on. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@zainal-abidin-assegaf
Copy link

Describe the bug
A clear and concise description of what the bug is.

Failed to get auth credentials: open /root/.pixie/auth.json: too many open files

To Reproduce
Steps to reproduce the behavior:

  1. px deploy

Expected behavior
A clear and concise description of what you expected to happen.

pixie installed successfully

Screenshots
If applicable, add screenshots to help explain your problem.

core@manager-01 ~ $ sudo px deploy
Pixie CLI

Running Cluster Checks:
 ✔    Kernel version > 4.14.0 
 ✔    Cluster type is supported 
 ✔    K8s version > 1.16.0 
 ✔    Kubectl > 1.10.0 is present 
 ✔    User can create namespace 
 ✕    Cluster type is in list of known supported types  ERR: Cluster type is not in list of known supported cluster types. Please see: https://docs.px.dev/installing-pixie/requirements/
Some cluster checks failed. Pixie may not work properly on your cluster. Continue with deploy? (y/n) [y] : y
Installing Vizier version: 0.12.9
Generating YAMLs for Pixie
Deploying Pixie to the following cluster: kubernetes-the-hard-way
Is the cluster correct? (y/n) [y] : y
Found 7 nodes
 ✔    Installing OLM CRDs 
 ✔    Deploying OLM 
 ✔    Deploying Pixie OLM Namespace 
 ✔    Installing Vizier CRD 
 ✔    Deploying OLM Catalog 
 ✔    Deploying OLM Subscription 
 ✔    Creating namespace 
 ✔    Deploying Vizier 
 ✔    Waiting for Cloud Connector to come online 
Waiting for Pixie to pass healthcheck
 ✔    Wait for PEMs/Kelvin 
 ⠹    Wait for healthcheck 
Failed to get auth credentials: open /root/.pixie/auth.json: too many open files

Logs
Please attach the logs by running the following command:

Name:               worker-001.bnpb.go.id
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker-001.bnpb.go.id
                    kubernetes.io/os=linux
                    label=minion
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"rook-ceph.cephfs.csi.ceph.com":"worker-001.bnpb.go.id","rook-ceph.rbd.csi.ceph.com":"worker-001.bnpb.go.id"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 21 Feb 2022 12:48:17 +0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker-001.bnpb.go.id
  AcquireTime:     <unset>
  RenewTime:       Thu, 15 Dec 2022 13:07:14 +0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 15 Dec 2022 13:02:35 +0700   Mon, 12 Dec 2022 02:02:43 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 15 Dec 2022 13:02:35 +0700   Mon, 12 Dec 2022 02:02:43 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 15 Dec 2022 13:02:35 +0700   Mon, 12 Dec 2022 02:02:43 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 15 Dec 2022 13:02:35 +0700   Mon, 12 Dec 2022 02:02:46 +0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.207.224
  Hostname:    worker-001.bnpb.go.id
Capacity:
  cpu:                8
  ephemeral-storage:  505527372Ki
  hugepages-2Mi:      0
  memory:             16383340Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  465894025264
  hugepages-2Mi:      0
  memory:             16280940Ki
  pods:               110
System Info:
  Machine ID:                      e0635838c9144f4fb5ccc31b8e5e2f1e
  System UUID:                     4220591d-21b6-ae6f-f712-2e8c6f439004
  Boot ID:                         f686d433-406a-4d58-8132-99e091f179d9
  Kernel Version:                  5.15.77-flatcar
  OS Image:                        Flatcar Container Linux by Kinvolk 3374.2.1 (Oklo)
  Operating System:                linux
  Architecture:                    amd64
  Container Runtime Version:       containerd://1.6.8
  Kubelet Version:                 v1.23.4
  Kube-Proxy Version:              v1.23.4
PodCIDR:                           10.244.1.0/24
PodCIDRs:                          10.244.1.0/24
Non-terminated Pods:               (18 in total)
  Namespace                        Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                        ----                                                               ------------  ----------  ---------------  -------------  ---
  default                          dnstools-66b47fd85b-kj8kd                                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         66d
  default                          pg-single-0                                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         192d
  inarisk                          inariskapiv3-6f85d8c9cb-bgtq8                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  inarisk                          inariskwebv4-66f488ffd4-fwbbn                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         236d
  inarisk                          pushalert-76848556c9-tlhkw                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         181d
  kube-system                      coredns-7cdff66f64-nv88n                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     181d
  kube-system                      fluentd-fvj62                                                      100m (1%)     0 (0%)      200Mi (1%)       200Mi (1%)     5d20h
  kube-system                      metrics-server-847dcc659d-7hh56                                    100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         181d
  kubernetes-dashboard             kubernetes-dashboard-887c5ff5-4jsn2                                100m (1%)     2 (25%)     200Mi (1%)       200Mi (1%)     181d
  metallb-system                   speaker-v7fhd                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  nfs-subdir-external-provisioner  nfs-subdir-external-provisioner-858494fb9-hqcpb                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         195d
  platform                         signoz-clickhouse-operator-5c5b6cbfd6-dzkv6                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         24h
  platform                         signoz-k8s-infra-otel-agent-7pxft                                  100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  rook-ceph                        csi-cephfsplugin-mdtl7                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                        csi-rbdplugin-84zpx                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                        rook-ceph-crashcollector-worker-001.bnpb.go.id-5f9798f8dd-c78sb    0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
  rook-ceph                        rook-ceph-mds-myfs-b-545bb57f8c-qmxg5                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
  rook-ceph                        rook-ceph-osd-2-95b6dd85c-dfdnt                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         236d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                500m (6%)   2 (25%)
  memory             770Mi (4%)  570Mi (3%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>


Name:               worker-002.bnpb.go.id
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker-002.bnpb.go.id
                    kubernetes.io/os=linux
                    label=minion
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"rook-ceph.cephfs.csi.ceph.com":"worker-002.bnpb.go.id","rook-ceph.rbd.csi.ceph.com":"worker-002.bnpb.go.id"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 21 Feb 2022 12:48:17 +0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker-002.bnpb.go.id
  AcquireTime:     <unset>
  RenewTime:       Thu, 15 Dec 2022 13:07:20 +0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 15 Dec 2022 13:02:37 +0700   Mon, 12 Dec 2022 19:53:05 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 15 Dec 2022 13:02:37 +0700   Mon, 12 Dec 2022 19:53:05 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 15 Dec 2022 13:02:37 +0700   Mon, 12 Dec 2022 19:53:05 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 15 Dec 2022 13:02:37 +0700   Mon, 12 Dec 2022 19:53:05 +0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.207.225
  Hostname:    worker-002.bnpb.go.id
Capacity:
  cpu:                8
  ephemeral-storage:  505527372Ki
  hugepages-2Mi:      0
  memory:             16383340Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  465894025264
  hugepages-2Mi:      0
  memory:             16280940Ki
  pods:               110
System Info:
  Machine ID:                 a41aa7f18a6d4ab3afbcdf3c4c505a28
  System UUID:                422092f0-29e9-4e4d-5d5a-91c45a4546c7
  Boot ID:                    ef8f905b-ddb7-42be-af83-035c2f14bdec
  Kernel Version:             5.15.77-flatcar
  OS Image:                   Flatcar Container Linux by Kinvolk 3374.2.1 (Oklo)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.8
  Kubelet Version:            v1.23.4
  Kube-Proxy Version:         v1.23.4
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
Non-terminated Pods:          (15 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  inarisk                     inariskapiv3-6f85d8c9cb-kzrxv                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  inarisk                     inariskwebv4-66f488ffd4-4kbgs                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         252d
  inarisk                     pushalert-76848556c9-86fn2                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         181d
  kube-system                 coredns-7cdff66f64-thf59                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     292d
  kube-system                 fluentd-pj2wn                                                      100m (1%)     0 (0%)      200Mi (1%)       200Mi (1%)     5d20h
  kube-system                 metrics-server-847dcc659d-pz4w2                                    100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         292d
  metallb-system              speaker-htgvj                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  platform                    signoz-k8s-infra-otel-agent-hnr8v                                  100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  rook-ceph                   csi-cephfsplugin-96krw                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   csi-cephfsplugin-provisioner-5dc9cbcc87-9772c                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   csi-rbdplugin-677fw                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   csi-rbdplugin-provisioner-58f584754c-59pcj                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   rook-ceph-crashcollector-worker-002.bnpb.go.id-6c9dc78f6f-52b2r    0 (0%)        0 (0%)      0 (0%)           0 (0%)         154d
  rook-ceph                   rook-ceph-mon-a-576f6999d-k2bqb                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   rook-ceph-osd-0-696b845d95-k9tcg                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                400m (5%)   0 (0%)
  memory             570Mi (3%)  370Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>


Name:               worker-003.bnpb.go.id
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker-003.bnpb.go.id
                    kubernetes.io/os=linux
                    label=minion
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"rook-ceph.cephfs.csi.ceph.com":"worker-003.bnpb.go.id","rook-ceph.rbd.csi.ceph.com":"worker-003.bnpb.go.id"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 21 Feb 2022 12:48:18 +0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker-003.bnpb.go.id
  AcquireTime:     <unset>
  RenewTime:       Thu, 15 Dec 2022 13:07:11 +0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 15 Dec 2022 13:04:44 +0700   Mon, 12 Dec 2022 02:06:26 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 15 Dec 2022 13:04:44 +0700   Mon, 12 Dec 2022 02:06:26 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 15 Dec 2022 13:04:44 +0700   Mon, 12 Dec 2022 02:06:26 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 15 Dec 2022 13:04:44 +0700   Mon, 12 Dec 2022 02:06:26 +0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.207.226
  Hostname:    worker-003.bnpb.go.id
Capacity:
  cpu:                8
  ephemeral-storage:  505527372Ki
  hugepages-2Mi:      0
  memory:             16383340Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  465894025264
  hugepages-2Mi:      0
  memory:             16280940Ki
  pods:               110
System Info:
  Machine ID:                 5eaa3155acb148748193ec8904e96b73
  System UUID:                42208c42-2f1c-080c-5555-374d6e2f462a
  Boot ID:                    4ac4215d-0592-45d4-88e3-7a21e47f0f82
  Kernel Version:             5.15.77-flatcar
  OS Image:                   Flatcar Container Linux by Kinvolk 3374.2.1 (Oklo)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.8
  Kubelet Version:            v1.23.4
  Kube-Proxy Version:         v1.23.4
PodCIDR:                      10.244.2.0/24
PodCIDRs:                     10.244.2.0/24
Non-terminated Pods:          (15 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  inarisk                     inariskapiv3-6f85d8c9cb-s9msf                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  inarisk                     inariskwebv4-66f488ffd4-2rhwx                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         252d
  inarisk                     pushalert-76848556c9-v9bs4                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         181d
  kube-system                 coredns-7cdff66f64-mxmpv                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     292d
  kube-system                 fluentd-t7gtb                                                      100m (1%)     0 (0%)      200Mi (1%)       200Mi (1%)     5d20h
  kube-system                 metrics-server-847dcc659d-28psp                                    100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         292d
  metallb-system              speaker-vpgjx                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  platform                    signoz-k8s-infra-otel-agent-ll7r7                                  100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  rook-ceph                   csi-cephfsplugin-lfnw6                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   csi-cephfsplugin-provisioner-5dc9cbcc87-pbkbs                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         265d
  rook-ceph                   csi-rbdplugin-lrxbh                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   csi-rbdplugin-provisioner-58f584754c-k7gzt                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   rook-ceph-crashcollector-worker-003.bnpb.go.id-6f98845988-j672d    0 (0%)        0 (0%)      0 (0%)           0 (0%)         171d
  rook-ceph                   rook-ceph-mon-b-6995c89687-vm7xk                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
  rook-ceph                   rook-ceph-osd-1-6f49d948b4-4vzs8                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         296d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                400m (5%)   0 (0%)
  memory             570Mi (3%)  370Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>


Name:               worker-004.bnpb.go.id
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker-004.bnpb.go.id
                    kubernetes.io/os=linux
                    label=minion
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"rook-ceph.cephfs.csi.ceph.com":"worker-004.bnpb.go.id","rook-ceph.rbd.csi.ceph.com":"worker-004.bnpb.go.id"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 26 Feb 2022 10:23:01 +0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker-004.bnpb.go.id
  AcquireTime:     <unset>
  RenewTime:       Thu, 15 Dec 2022 13:07:16 +0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 15 Dec 2022 13:03:19 +0700   Thu, 15 Dec 2022 12:47:58 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 15 Dec 2022 13:03:19 +0700   Thu, 15 Dec 2022 12:47:58 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 15 Dec 2022 13:03:19 +0700   Thu, 15 Dec 2022 12:47:58 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 15 Dec 2022 13:03:19 +0700   Thu, 15 Dec 2022 12:47:58 +0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.207.227
  Hostname:    worker-004.bnpb.go.id
Capacity:
  cpu:                8
  ephemeral-storage:  505527372Ki
  hugepages-2Mi:      0
  memory:             16383340Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  465894025264
  hugepages-2Mi:      0
  memory:             16280940Ki
  pods:               110
System Info:
  Machine ID:                 001a36240b454bddbeb43f2453a95815
  System UUID:                4220f23b-7104-0df4-233f-76e5cca48d01
  Boot ID:                    4707429d-4809-4c33-89ef-4c65d75e52db
  Kernel Version:             5.15.77-flatcar
  OS Image:                   Flatcar Container Linux by Kinvolk 3374.2.1 (Oklo)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.8
  Kubelet Version:            v1.23.4
  Kube-Proxy Version:         v1.23.4
PodCIDR:                      10.244.5.0/24
PodCIDRs:                     10.244.5.0/24
Non-terminated Pods:          (15 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  inarisk                     inariskapiv3-6f85d8c9cb-cbm89                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  keydb                       keydb-1                                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         66d
  kube-system                 fluentd-psxq7                                                      100m (1%)     0 (0%)      200Mi (1%)       200Mi (1%)     5d20h
  metallb-system              speaker-68fkh                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  olm                         olm-operator-7fc4465885-487w2                                      10m (0%)      0 (0%)      160Mi (1%)       0 (0%)         27m
  platform                    chi-signoz-clickhouse-cluster-0-0-0                                100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         24h
  platform                    signoz-frontend-df7d4c58b-n5tkj                                    100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  platform                    signoz-k8s-infra-otel-agent-gtn6d                                  100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  platform                    signoz-k8s-infra-otel-deployment-7fd6c645fd-762cd                  100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  platform                    signoz-query-service-0                                             100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  pushalert-dev               pushalert-dev-7fb55475bf-ml9kf                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         43d
  rook-ceph                   csi-cephfsplugin-kzr79                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                   csi-rbdplugin-n9vr9                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                   rook-ceph-crashcollector-worker-004.bnpb.go.id-78ff95b96c-q762b    0 (0%)        0 (0%)      0 (0%)           0 (0%)         24m
  rook-ceph                   rook-ceph-osd-5-84f4846999-ct7hz                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         66d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                610m (7%)   0 (0%)
  memory             960Mi (6%)  200Mi (1%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type    Reason                   Age                   From     Message
  ----    ------                   ----                  ----     -------
  Normal  NodeHasSufficientMemory  19m (x12 over 3d11h)  kubelet  Node worker-004.bnpb.go.id status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    19m (x12 over 3d11h)  kubelet  Node worker-004.bnpb.go.id status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     19m (x12 over 3d11h)  kubelet  Node worker-004.bnpb.go.id status is now: NodeHasSufficientPID
  Normal  NodeReady                19m (x4 over 3d11h)   kubelet  Node worker-004.bnpb.go.id status is now: NodeReady


Name:               worker-005.bnpb.go.id
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker-005.bnpb.go.id
                    kubernetes.io/os=linux
                    label=minion
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"rook-ceph.cephfs.csi.ceph.com":"worker-005.bnpb.go.id","rook-ceph.rbd.csi.ceph.com":"worker-005.bnpb.go.id"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 26 Feb 2022 10:23:01 +0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker-005.bnpb.go.id
  AcquireTime:     <unset>
  RenewTime:       Thu, 15 Dec 2022 13:07:17 +0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 15 Dec 2022 13:03:43 +0700   Mon, 12 Dec 2022 02:02:51 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 15 Dec 2022 13:03:43 +0700   Mon, 12 Dec 2022 02:02:51 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 15 Dec 2022 13:03:43 +0700   Mon, 12 Dec 2022 02:02:51 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 15 Dec 2022 13:03:43 +0700   Mon, 12 Dec 2022 02:02:51 +0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.207.228
  Hostname:    worker-005.bnpb.go.id
Capacity:
  cpu:                8
  ephemeral-storage:  505527372Ki
  hugepages-2Mi:      0
  memory:             16383372Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  465894025264
  hugepages-2Mi:      0
  memory:             16280972Ki
  pods:               110
System Info:
  Machine ID:                      94b35298027547de9ee8f7a6f23e0d6b
  System UUID:                     42209598-25aa-7810-a613-09531722b8b3
  Boot ID:                         fe5b26b8-2d92-494f-a7a6-e5e078638fd0
  Kernel Version:                  5.15.77-flatcar
  OS Image:                        Flatcar Container Linux by Kinvolk 3374.2.1 (Oklo)
  Operating System:                linux
  Architecture:                    amd64
  Container Runtime Version:       containerd://1.6.8
  Kubelet Version:                 v1.23.4
  Kube-Proxy Version:              v1.23.4
PodCIDR:                           10.244.6.0/24
PodCIDRs:                          10.244.6.0/24
Non-terminated Pods:               (19 in total)
  Namespace                        Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                        ----                                                               ------------  ----------  ---------------  -------------  ---
  inarisk                          inariskapiv3-6f85d8c9cb-kmgzj                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  inarisk                          inariskwebv4-66f488ffd4-gk4s9                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         66d
  inarisk                          pushalert-76848556c9-47gv6                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         66d
  kube-system                      coredns-7cdff66f64-q88dh                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     66d
  kube-system                      fluentd-mls5q                                                      100m (1%)     0 (0%)      200Mi (1%)       200Mi (1%)     5d20h
  kube-system                      metrics-server-847dcc659d-tthqx                                    100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         66d
  metallb-system                   speaker-mgwtz                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         268d
  nfs-subdir-external-provisioner  nfs-subdir-external-provisioner-858494fb9-7lnl9                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         66d
  platform                         signoz-alertmanager-0                                              100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  platform                         signoz-k8s-infra-otel-agent-j62j4                                  100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  platform                         signoz-zookeeper-0                                                 250m (3%)     0 (0%)      256Mi (1%)       0 (0%)         24h
  registry                         registry-7557646d79-ks9kt                                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         138d
  rook-ceph                        csi-cephfsplugin-88j46                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                        csi-rbdplugin-2gkvj                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                        rook-ceph-crashcollector-worker-005.bnpb.go.id-55db9bdf9d-x4j95    0 (0%)        0 (0%)      0 (0%)           0 (0%)         19d
  rook-ceph                        rook-ceph-mgr-a-784dfd5b9-jbt7c                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
  rook-ceph                        rook-ceph-mon-d-d5464fbff-4b9f8                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         143d
  rook-ceph                        rook-ceph-osd-6-7f77f58876-6w9jf                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         143d
  rook-ceph                        rook-ceph-tools-ccf46c4f7-vd5tv                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                750m (9%)   0 (0%)
  memory             926Mi (5%)  370Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>


Name:               worker-006.bnpb.go.id
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker-006.bnpb.go.id
                    kubernetes.io/os=linux
                    label=minion
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"rook-ceph.cephfs.csi.ceph.com":"worker-006.bnpb.go.id","rook-ceph.rbd.csi.ceph.com":"worker-006.bnpb.go.id"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 26 Feb 2022 10:23:03 +0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker-006.bnpb.go.id
  AcquireTime:     <unset>
  RenewTime:       Thu, 15 Dec 2022 13:07:19 +0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 15 Dec 2022 13:07:17 +0700   Mon, 12 Dec 2022 02:02:48 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 15 Dec 2022 13:07:17 +0700   Mon, 12 Dec 2022 02:02:48 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 15 Dec 2022 13:07:17 +0700   Mon, 12 Dec 2022 02:02:48 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 15 Dec 2022 13:07:17 +0700   Mon, 12 Dec 2022 02:02:48 +0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.207.229
  Hostname:    worker-006.bnpb.go.id
Capacity:
  cpu:                8
  ephemeral-storage:  505527372Ki
  hugepages-2Mi:      0
  memory:             16383340Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  465894025264
  hugepages-2Mi:      0
  memory:             16280940Ki
  pods:               110
System Info:
  Machine ID:                 d2e527c059b343c99812dd7d4945dfb5
  System UUID:                4220d9de-5fdb-6bad-d03a-4f12f8e77262
  Boot ID:                    036b6e7b-c762-47f5-90e0-01c09f62f0bc
  Kernel Version:             5.15.77-flatcar
  OS Image:                   Flatcar Container Linux by Kinvolk 3374.2.1 (Oklo)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.8
  Kubelet Version:            v1.23.4
  Kube-Proxy Version:         v1.23.4
PodCIDR:                      10.244.3.0/24
PodCIDRs:                     10.244.3.0/24
Non-terminated Pods:          (19 in total)
  Namespace                   Name                                                              CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                              ------------  ----------  ---------------  -------------  ---
  default                     dnstools-66b47fd85b-5tb9x                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         143d
  inarisk                     inariskapiv3-6f85d8c9cb-4xmnt                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  inarisk                     inariskwebv4-66f488ffd4-rt2rj                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         143d
  inarisk                     pushalert-76848556c9-b6d9k                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         143d
  keydb                       keydb-2                                                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         181d
  kube-system                 coredns-7cdff66f64-q62gd                                          100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     143d
  kube-system                 fluentd-k8zvh                                                     100m (1%)     0 (0%)      200Mi (1%)       200Mi (1%)     5d20h
  kube-system                 metrics-server-847dcc659d-h6wj8                                   100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         143d
  kubernetes-dashboard        kubernetes-dashboard-887c5ff5-js42r                               100m (1%)     2 (25%)     200Mi (1%)       200Mi (1%)     66d
  metallb-system              speaker-q59qq                                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  olm                         catalog-operator-57f7fd7d76-w2x64                                 10m (0%)      0 (0%)      80Mi (0%)        0 (0%)         27m
  platform                    signoz-k8s-infra-otel-agent-bdh7t                                 100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  platform                    signoz-otel-collector-7b8687d8d4-nknbk                            100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         24h
  platform                    signoz-otel-collector-metrics-68784f5874-glwww                    100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  rook-ceph                   csi-cephfsplugin-9tdl9                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                   csi-rbdplugin-m6259                                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                   rook-ceph-crashcollector-worker-006.bnpb.go.id-f6dd9c899-mlh72    0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
  rook-ceph                   rook-ceph-mds-myfs-a-65867d95f7-q2pt2                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
  rook-ceph                   rook-ceph-osd-4-b9c486c9d-wwbxw                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         181d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                710m (8%)    2 (25%)
  memory             1150Mi (7%)  570Mi (3%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:              <none>


Name:               worker-007.bnpb.go.id
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker-007.bnpb.go.id
                    kubernetes.io/os=linux
                    label=minion
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"rook-ceph.cephfs.csi.ceph.com":"worker-007.bnpb.go.id","rook-ceph.rbd.csi.ceph.com":"worker-007.bnpb.go.id"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 26 Feb 2022 10:22:58 +0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker-007.bnpb.go.id
  AcquireTime:     <unset>
  RenewTime:       Thu, 15 Dec 2022 13:07:15 +0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 15 Dec 2022 13:02:57 +0700   Mon, 12 Dec 2022 02:11:46 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 15 Dec 2022 13:02:57 +0700   Mon, 12 Dec 2022 02:11:46 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 15 Dec 2022 13:02:57 +0700   Mon, 12 Dec 2022 02:11:46 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 15 Dec 2022 13:02:57 +0700   Mon, 12 Dec 2022 02:11:46 +0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.207.230
  Hostname:    worker-007.bnpb.go.id
Capacity:
  cpu:                8
  ephemeral-storage:  505527372Ki
  hugepages-2Mi:      0
  memory:             16383372Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  465894025264
  hugepages-2Mi:      0
  memory:             16280972Ki
  pods:               110
System Info:
  Machine ID:                      a1996fea1a9e4ea2bddabfc58b96b811
  System UUID:                     4220b617-2006-0ad1-2a1c-edf4c8254f2e
  Boot ID:                         31fa73b4-ac41-42be-b248-efd6aa3be6dc
  Kernel Version:                  5.15.77-flatcar
  OS Image:                        Flatcar Container Linux by Kinvolk 3374.2.1 (Oklo)
  Operating System:                linux
  Architecture:                    amd64
  Container Runtime Version:       containerd://1.6.8
  Kubelet Version:                 v1.23.4
  Kube-Proxy Version:              v1.23.4
PodCIDR:                           10.244.4.0/24
PodCIDRs:                          10.244.4.0/24
Non-terminated Pods:               (21 in total)
  Namespace                        Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                        ----                                                               ------------  ----------  ---------------  -------------  ---
  default                          dnstools-66b47fd85b-dmfrb                                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         267d
  inarisk                          inariskapiv3-6f85d8c9cb-bhdnv                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d18h
  inarisk                          inariskwebv4-66f488ffd4-5knj4                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         66d
  inarisk                          inariskwebv4-66f488ffd4-zw6rn                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         252d
  inarisk                          pushalert-76848556c9-hg85s                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         181d
  keydb                            keydb-0                                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         271d
  kube-system                      coredns-7cdff66f64-9jvrp                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     181d
  kube-system                      coredns-7cdff66f64-l9hmq                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     292d
  kube-system                      fluentd-r924n                                                      100m (1%)     0 (0%)      200Mi (1%)       200Mi (1%)     5d20h
  kube-system                      metrics-server-847dcc659d-qmzjw                                    100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         292d
  kube-system                      metrics-server-847dcc659d-qpw8n                                    100m (1%)     0 (0%)      200Mi (1%)       0 (0%)         181d
  kubernetes-dashboard             kubernetes-dashboard-887c5ff5-sdb8c                                100m (1%)     2 (25%)     200Mi (1%)       200Mi (1%)     268d
  metallb-system                   controller-57fd9c5bb-s6gwp                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         181d
  metallb-system                   speaker-67fxf                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  nfs-subdir-external-provisioner  nfs-subdir-external-provisioner-858494fb9-7dvtw                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         143d
  platform                         signoz-k8s-infra-otel-agent-f6zwr                                  100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         24h
  rook-ceph                        csi-cephfsplugin-xrx68                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                        csi-rbdplugin-wbldl                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
  rook-ceph                        rook-ceph-crashcollector-worker-007.bnpb.go.id-5f884c8888-z26v4    0 (0%)        0 (0%)      0 (0%)           0 (0%)         213d
  rook-ceph                        rook-ceph-operator-7d9bf8cccf-vbjkr                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         265d
  rook-ceph                        rook-ceph-osd-3-fbfc9597c-72xnd                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         292d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                700m (8%)    2 (25%)
  memory             1040Mi (6%)  740Mi (4%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:              <none>

App information (please complete the following information):

  • Pixie version: v0.7.17
  • K8s cluster version: v1.23.4

Additional context
Add any other context about the problem here.
@aimichelle aimichelle added kind/support Categorizes issue or PR as a support question. area/deployment Issues replated to deployments triage/needs-information Indicates an issue needs more information in order to work on it. triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Dec 15, 2022
@aimichelle
Copy link
Member

Hey @4ss3g4f ! We've seen this issue with our CLI before (#312). Would you mind increasing the file descriptor limit temporarily to see if that helps? ulimit -n 10240

@zainal-abidin-assegaf
Copy link
Author

Hi @aimichelle
Still error,

core@localrepo ~ $ sudo px deploy
Pixie CLI

Running Cluster Checks:
 ✔    Kernel version > 4.14.0 
 ✔    Cluster type is supported 
 ✔    K8s version > 1.16.0 
 ✔    Kubectl > 1.10.0 is present 
 ✔    User can create namespace 
 ✕    Cluster type is in list of known supported types  ERR: Cluster type is not in list of known supported cluster types. Please see: https://docs.px.dev/installing-pixie/requirements/
Some cluster checks failed. Pixie may not work properly on your cluster. Continue with deploy? (y/n) [y] : y
Installing Vizier version: 0.12.9
Generating YAMLs for Pixie
Deploying Pixie to the following cluster: kubernetes-the-hard-way
Is the cluster correct? (y/n) [y] : y
Found 1 nodes
 ✔    Installing OLM CRDs 
 ✔    Deploying OLM 
 ✔    Deploying Pixie OLM Namespace 
 ✔    Installing Vizier CRD 
 ✔    Deploying OLM Catalog 
 ✔    Deploying OLM Subscription 
 ✔    Creating namespace 
 ✔    Deploying Vizier 
 ✔    Waiting for Cloud Connector to come online 
Waiting for Pixie to pass healthcheck
 ✔    Wait for PEMs/Kelvin 
 ✔    Wait for PEMs/Kelvin 
 ✕    Wait for healthcheck  ERR: timeout waiting for healthcheck  (it is possible that Pixie stabilized after the healthcheck timeout. To check if Pixie successfully deployed, run `px debug pods`)
Failed Pixie healthcheck error=timeout waiting for healthcheck  (it is possible that Pixie stabilized after the healthcheck timeout. To check if Pixie successfully deployed, run `px debug pods`)
core@localrepo ~ $ sudo px debug pods
Pixie CLI
Cluster ID : 02aeec08-6cf0-4e4d-b5c8-5d506ad15029
Could not fetch Vizier pods error=context deadline exceeded

@zainal-abidin-assegaf
Copy link
Author 

core@localrepo ~ $ sudo kubectl get all -n px-operator
NAME                                                                  READY   STATUS      RESTARTS   AGE
pod/5ff7d47213a8875e3f1827d728b149498a0fdef08ba74499866d7d51a4qpwsz   0/1     Completed   0          23m
pod/pixie-operator-index-k28cr                                        1/1     Running     0          23m
pod/vizier-operator-88fbf7f87-hvm5j                                   1/1     Running     0          22m

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/pixie-operator-index   ClusterIP   10.100.206.246   <none>        50051/TCP   23m

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vizier-operator   1/1     1            1           22m

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/vizier-operator-88fbf7f87   1         1         1       22m

NAME                                                                        COMPLETIONS   DURATION   AGE
job.batch/5ff7d47213a8875e3f1827d728b149498a0fdef08ba74499866d7d51a4b0147   1/1           35s        23m
core@localrepo ~ $ sudo px debug pods
Pixie CLI
Cluster ID : 02aeec08-6cf0-4e4d-b5c8-5d506ad15029
Could not fetch Vizier pods error=context deadline exceeded
core@localrepo ~ $ sudo kubectl logs vizier-operator-88fbf7f87-hvm5j -n px-operator
time="2022-12-16T08:40:22Z" level=info msg="Starting manager"
time="2022-12-16T08:40:23Z" level=info msg="Reconciling Vizier..." req=pl/pixie
time="2022-12-16T08:40:23Z" level=info msg="Creating a new vizier instance"
time="2022-12-16T08:40:23Z" level=info msg="Starting a vizier deploy"
time="2022-12-16T08:40:23Z" level=info msg="Deploying Vizier configs and secrets"
time="2022-12-16T08:40:23Z" level=info msg="Generating certs"
time="2022-12-16T08:40:34Z" level=info msg="Deploying NATS"
time="2022-12-16T08:40:34Z" level=info msg="Deploying Vizier"
time="2022-12-16T08:41:54Z" level=info msg="Vizier deploy is complete"
time="2022-12-16T08:41:54Z" level=info msg="Reconciling Vizier..." req=pl/pixie
time="2022-12-16T08:41:54Z" level=info msg="Updating Vizier..."
time="2022-12-16T08:41:54Z" level=info msg="Checksums matched, no need to reconcile"
time="2022-12-16T08:41:54Z" level=info msg="Reconciling Vizier..." req=pl/pixie
time="2022-12-16T08:41:54Z" level=info msg="Updating Vizier..."
time="2022-12-16T08:41:54Z" level=info msg="Checksums matched, no need to reconcile"
time="2022-12-16T08:42:14Z" level=info msg="Reconciling Vizier..." req=pl/pixie
time="2022-12-16T08:42:14Z" level=info msg="Updating Vizier..."
time="2022-12-16T08:42:14Z" level=info msg="Checksums matched, no need to reconcile"
time="2022-12-16T08:43:54Z" level=info msg="Reconciling Vizier..." req=pl/pixie
time="2022-12-16T08:43:54Z" level=info msg="Updating Vizier..."
time="2022-12-16T08:43:54Z" level=info msg="Checksums matched, no need to reconcile"
time="2022-12-16T08:48:34Z" level=info msg="Reconciling Vizier..." req=pl/pixie
time="2022-12-16T08:48:34Z" level=info msg="Updating Vizier..."
time="2022-12-16T08:48:34Z" level=info msg="Checksums matched, no need to reconcile"
core@localrepo ~ $

@zainal-abidin-assegaf
Copy link
Author

is pixie used liveness and readiness probe for health check ??
we can redirect liveness and readiness probe to have successful installation

@scomri
Copy link

scomri commented Feb 27, 2023 •
edited
Loading

I also can't get px deploy to finish running. Always crashes after waiting for health checks.
Tried to fix with setting ulimit to 10240 and deleting auth.json - still without success.

Pixie logs:

[cloudshell-user@ip-10-2-93-38 ~]$ px collect-logs
Pixie CLI
WARN[0001] Failed to log pod: kelvin-76cd6f549c-zd5cn    error="container \"app\" in pod \"kelvin-76cd6f549c-zd5cn\" is waiting to start: PodInitializing"
WARN[0001] Failed to log pod: vizier-pem-djq78           error="container \"pem\" in pod \"vizier-pem-djq78\" is waiting to start: PodInitializing"
WARN[0001] Failed to log pod: vizier-pem-s8mhd           error="container \"pem\" in pod \"vizier-pem-s8mhd\" is waiting to start: PodInitializing"
WARN[0001] Failed to log pod: vizier-pem-xwhwf           error="container \"pem\" in pod \"vizier-pem-xwhwf\" is waiting to start: PodInitializing"
WARN[0002] Failed to log pod: vizier-query-broker-798754d8d9-9j9cr  error="container \"app\" in pod \"vizier-query-broker-798754d8d9-9j9cr\" is waiting to start: PodInitializing"

@jberryman jberryman mentioned this issue Sep 4, 2024
Closed
@jberryman
Copy link

FWIW this resolved the mystery issue I was having: #2006 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/deployment Issues replated to deployments kind/support Categorizes issue or PR as a support question. triage/accepted Indicates an issue or PR is ready to be actively worked on. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jberryman @aimichelle @zainal-abidin-assegaf @scomri