Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When expanding the queryNode pod, other pods terminate and restart #6414

Closed
3 tasks
ThreadDao opened this issue Jul 9, 2021 · 2 comments
Closed
3 tasks
Assignees
Labels
do-not-merge/hold kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone

Comments

@ThreadDao
Copy link
Contributor

Please state your issue using the following template and, most importantly, in English.

Describe the bug

  1. Before expanding queryNode pod:
cluster-zong-etcd-0                               1/1     Running   0          14m
cluster-zong-milvus-datacoord-546dc69b66-46qgr    1/1     Running   0          2m14s
cluster-zong-milvus-datanode-54c658885c-kcdxj     1/1     Running   0          2m14s
cluster-zong-milvus-indexcoord-5d767bbc57-79b8l   1/1     Running   0          2m14s
cluster-zong-milvus-indexnode-585878d78b-tfkp4    1/1     Running   0          2m14s
cluster-zong-milvus-proxy-7ccd448897-hbmh4        1/1     Running   0          2m14s
cluster-zong-milvus-pulsar-7fff85f66f-r9cvk       1/1     Running   0          2m14s
cluster-zong-milvus-querycoord-64fdfcd5c6-pchjq   1/1     Running   0          2m14s
cluster-zong-milvus-querynode-6f8c765596-bcd99    1/1     Running   0          2m13s
cluster-zong-milvus-rootcoord-98bf769c8-ckkc8     1/1     Running   0          2m14s
cluster-zong-minio-d46669cdc-xc6c8                1/1     Running   0          14m
  1. During expanding queryNode pod:
helm upgrade --install --wait --timeout 180s --set image.all.repository=milvusdb/milvus-dev --set image.all.tag=master-latest --set cluster.enabled=true --set service.type=LoadBalancer --set queryNode.replicas=3 cluster-zong milvus/milvus
cluster-zong-etcd-0                               1/1     Running             0          14m
cluster-zong-milvus-datacoord-546dc69b66-46qgr    1/1     Terminating         0          2m24s
cluster-zong-milvus-datacoord-7959b6f4b4-wh567    1/1     Running             0          7s
cluster-zong-milvus-datanode-54c658885c-kcdxj     1/1     Terminating         0          2m24s
cluster-zong-milvus-datanode-68fb4b88c9-h4z56     1/1     Running             0          7s
cluster-zong-milvus-indexcoord-5d767bbc57-79b8l   1/1     Terminating         0          2m24s
cluster-zong-milvus-indexcoord-7559685999-r7llc   1/1     Running             0          7s
cluster-zong-milvus-indexnode-557cf76b76-w5dhp    1/1     Running             0          7s
cluster-zong-milvus-indexnode-585878d78b-tfkp4    1/1     Terminating         0          2m24s
cluster-zong-milvus-proxy-5757f56d7f-t5rv9        0/1     ContainerCreating   0          7s
cluster-zong-milvus-proxy-7ccd448897-hbmh4        1/1     Running             0          2m24s
cluster-zong-milvus-pulsar-67f899b688-h7kk6       1/1     Running             0          6s
cluster-zong-milvus-pulsar-7fff85f66f-r9cvk       1/1     Terminating         0          2m24s
cluster-zong-milvus-querycoord-64fdfcd5c6-pchjq   1/1     Running             0          2m24s
cluster-zong-milvus-querycoord-677c5975b8-5wzdc   0/1     ContainerCreating   0          6s
cluster-zong-milvus-querynode-6f7f4444cd-djg7c    0/1     ContainerCreating   0          5s
cluster-zong-milvus-querynode-6f8c765596-4p5sw    0/1     ContainerCreating   0          6s
cluster-zong-milvus-querynode-6f8c765596-bcd99    1/1     Running             0          2m23s
cluster-zong-milvus-querynode-6f8c765596-l2hr6    0/1     ContainerCreating   0          6s
cluster-zong-milvus-rootcoord-7fb4845968-mzgbf    0/1     ContainerCreating   0          5s
cluster-zong-milvus-rootcoord-98bf769c8-ckkc8     1/1     Running             0          2m24s
cluster-zong-minio-d46669cdc-xc6c8                1/1     Running             0          14m
  1. After expanding, the queryNode number is correct, but other pods also restart
cluster-zong-etcd-0                               1/1     Running   0          15m
cluster-zong-milvus-datacoord-7959b6f4b4-wh567    1/1     Running   0          44s
cluster-zong-milvus-datanode-68fb4b88c9-h4z56     1/1     Running   0          44s
cluster-zong-milvus-indexcoord-7559685999-r7llc   1/1     Running   0          44s
cluster-zong-milvus-indexnode-557cf76b76-w5dhp    1/1     Running   0          44s
cluster-zong-milvus-proxy-5757f56d7f-t5rv9        1/1     Running   0          44s
cluster-zong-milvus-pulsar-67f899b688-h7kk6       1/1     Running   0          43s
cluster-zong-milvus-querycoord-677c5975b8-5wzdc   1/1     Running   0          43s
cluster-zong-milvus-querynode-6f7f4444cd-djg7c    1/1     Running   0          42s
cluster-zong-milvus-querynode-6f7f4444cd-pm7q9    1/1     Running   0          30s
cluster-zong-milvus-querynode-6f7f4444cd-ssld2    1/1     Running   0          33s
cluster-zong-milvus-rootcoord-7fb4845968-mzgbf    1/1     Running   0          42s
cluster-zong-minio-d46669cdc-xc6c8                1/1     Running   0          15m

Steps/Code to reproduce behavior
Follow this guide to craft a minimal bug report. This helps us reproduce the issue you're having and resolve the issue more quickly.

Expected behavior
A clear and concise description of what you expected to happen.

Method of installation

  • Docker/cpu
  • Docker/gpu
  • Build from source

Environment details

  • Hardware/Software conditions (OS, CPU, GPU, Memory)

  • Milvus version (master or released version)
    GitCommit: 19fc42c

Configuration file
Settings you made in server_config.yaml or milvus.yaml

paste-file-content-here

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

@ThreadDao ThreadDao added the kind/bug Issues or changes related a bug label Jul 9, 2021
@ThreadDao ThreadDao added this to the 2.0-Backlog milestone Jul 9, 2021
@wxyucs wxyucs added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jul 13, 2021
@ThreadDao ThreadDao added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 15, 2021
@stale
Copy link

stale bot commented Aug 25, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale
Copy link

stale bot commented Sep 24, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Sep 24, 2021
@stale stale bot closed this as completed Oct 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/hold kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests

3 participants