Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Milvus fails to register components to etcd, which causes milvus not healthy #37402

Closed
1 task done
yanliang567 opened this issue Nov 4, 2024 · 5 comments
Closed
1 task done

[Bug]: Milvus fails to register components to etcd, which causes milvus not healthy #37402

yanliang567 opened this issue Nov 4, 2024 · 5 comments
Assignees
@yanliang567 @congqixia
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@yanliang567
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20241104-f54cf418-amd64
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):

Current Behavior

milvus is not running healthy

yanliang-mas-etcd-0                                               1/1     Running            0                 24m     10.104.25.193   4am-node30   <none>           <none>
yanliang-mas-milvus-standalone-5c44d54476-fdmbb                   0/1     Running            0                 22m     10.104.25.212   4am-node30   <none>           <none>
yanliang-mas-minio-5f56fd8764-p5s94                               1/1     Running            0                 24m     10.104.25.192   4am-node30   <none>           <none>

Expected Behavior

running healthy

Steps To Reproduce

delpey milvus with that tag

Milvus Log

logs from milvus:
image
image
logs from etcd:
image

Anything else?

No response
@yanliang567 yanliang567 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 4, 2024
@yanliang567 yanliang567 self-assigned this Nov 4, 2024
@yanliang567
Copy link
Contributor Author

/assign @congqixia
/unassign

@sre-ci-robot sre-ci-robot assigned congqixia and unassigned yanliang567 Nov 4, 2024
@yanliang567 yanliang567 added this to the 2.5.0 milestone Nov 4, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 4, 2024
@yanliang567 yanliang567 mentioned this issue Nov 4, 2024
Closed
1 task
@congqixia
Copy link
Contributor 

[INFO] [rootcoord/service.go:154] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]

Looks like the IP address returned from lib is an IPv6 address and not readable into session
working on why this happened

congqixia added a commit to congqixia/milvus that referenced this issue Nov 4, 2024
@congqixia
fix: Add IP address validation from paramtable
142d82d 
See also milvus-io#37404 milvus-io#37402

IP address in paramtable need validation and fail fast with reasonable
error message

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this issue Nov 4, 2024
Merged
@yanliang567
Copy link
Contributor Author

I can deploy milvus standalone successfully now with the pr #37418

@yanliang567 yanliang567 removed the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Nov 5, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Nov 5, 2024
@congqixia
fix: Add IP address validation from paramtable
c409ac2 
See also milvus-io#37404 milvus-io#37402

IP address in paramtable need validation and fail fast with reasonable
error message

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia
Copy link
Contributor

congqixia commented Nov 6, 2024 •
edited
Loading

last nightly run started successfully after #37445 merged, could you please verify
/assign @yanliang567

@yanliang567
Copy link
Contributor Author

verified

congqixia added a commit to congqixia/milvus that referenced this issue Nov 6, 2024
@congqixia
fix: Add IP address validation from paramtable
8158317 
See also milvus-io#37404 milvus-io#37402

IP address in paramtable need validation and fail fast with reasonable
error message

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Nov 7, 2024
@congqixia
fix: Add IP address validation from paramtable (#37416)
6325d02 
See also #37404 #37402

IP address in paramtable need validation and fail fast with reasonable
error message

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia added a commit to congqixia/milvus that referenced this issue Nov 7, 2024
@congqixia
fix: Add IP address validation from paramtable (milvus-io#37416)
000c1b2 
See also milvus-io#37404 milvus-io#37402

IP address in paramtable need validation and fail fast with reasonable
error message

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this issue Nov 7, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 11, 2024
@congqixia
fix: [2.4] Add IP address validation from paramtable (#37416) (#37500)
4f42611 
Cherry-pick from master
pr: #37416
See also #37404 #37402

IP address in paramtable need validation and fail fast with reasonable
error message

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yanliang567 yanliang567

@congqixia congqixia

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
2 participants
@yanliang567 @congqixia