[Feature]: Add retry for etcd operation

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

Milvus highly rely on etcd for meta operation.
Retry is need since failed to operate on etcd may cause server crash.
To be noticed, the CompareValueAndSwap, CompareVersionAndSwap may no be able to add retry in a no brainer style so we need to take care about it.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

[niebayes]

Is this issue still active? I'd like to take it.

[charleskakumanu]

Hi @xiaofan-luan
do we need to add retry for server start or for all operations?
added server start retry in #34208

[xiaofan-luan]

overall looks good to me, wait for ci pass