client: add a mechanism for various endpoint selection mode #4030

mitake · 2015-12-21T09:19:40Z

Current etcd client library chooses a default destination node from
every member of a cluster in a random manner. However, requests of
write and read (for consistent results) need to be forwarded to the
leader node as the nature of Raft algorithm. If the chosen node is a
follower, additional network traffic will be caused by the forwarding
from follower to leader.

Mainly for reducing the forward traffic, this commit adds a new
mechanism for various endpoint selection mode to the client library
which can be configured with client.Config.SelectionMode.

Currently, two modes are provided:

EndpointSelectionRandom: default, same to existing behavior (pick
a node in a random manner)
EndpointSelectionPrioritizeLeader: prioritize leader, for the above
purpose

I evaluated the effectiveness of the EndpointSelectionPrioritizeLeader
with 4 t1.micro instances of AWS (3 nodes for etcd cluster and 1 node
for etcd client). Client executes this simple benchmark
(https://github.com/mitake/etcd-things/tree/master/prioritize-leader-bench),
just writes 10000 keys. When SelectionMode == EndpointSelectionRandom
(default), the benchmark needed 1 min and 32.102 sec to finish. When
SelectionMode == EndpointSelectionPrioritizeLeader, the benchmark
needed 1 min 4.760 sec.

This PR also adds a new Leader() call to MembersAPI for the EndpointSelectionPrioritizeLeader mode.

gyuho · 2015-12-21T12:03:43Z

client/client.go

+type EndpointSelectionMode int
+
+const (
+	// EndpointSelectionDefault: pick an endpoint in a random manner


Please make comments full sentences as EndpointSelectionDefault is to pick an endpoint in a random manner.

mitake · 2015-12-22T02:35:52Z

@gyuho fixed the style of comments

gyuho · 2015-12-22T03:09:00Z

Thanks @mitake . I want to benchmark this function tomorrow, to double check in my side. Thanks for working on this.

xiang90 · 2015-12-22T05:02:49Z

client/client.go

+
+const (
+	// EndpointSelectionDefault is to pick an endpoint in a random manner.
+	EndpointSelectionDefault EndpointSelectionMode = iota


EndpointSelectionRandom?

xiang90 · 2015-12-22T05:10:43Z

Let's fix these style issues and remove the "unnecessary" feature. Thanks a lot.

mitake · 2015-12-22T06:00:42Z

@xiang90 Removed EndpointSelectionPreference and fixed style problems pointed, PTAL.

xiang90 · 2015-12-22T06:02:02Z

@mitake LGTM. @gyuho wants to double check the result so I would like to defer this to him. Thanks!

gyuho · 2015-12-22T06:05:08Z

Yeah, I just want to confirm that I see the same performance gain as you.
We will merge tomorrow. Thanks @mitake.

xiang90 · 2015-12-22T06:05:14Z

gofmt seems to be failing.

mitake · 2015-12-22T06:07:40Z

@xiang90 oops, sorry for that. Fixed it.

mitake · 2015-12-22T08:52:53Z

updated commit message

jonboulle · 2015-12-22T15:11:02Z

client/members.go

+
+func (l *membersAPIActionLeader) HTTPRequest(ep url.URL) *http.Request {
+	u := v2MembersURL(ep)
+	req, _ := http.NewRequest("GET", u.String()+"/leader", nil)


Please follow the style of the other functions: https://github.com/coreos/etcd/blob/eaaf98348ce433ff0a8ff94552649af215754df3/client/members.go#L216
I also suggest splitting out const leaderSuffix = "leader".

gyuho · 2015-12-22T23:55:27Z

@xiang90 @jonboulle I tested this in a simple 3-member cluster.

With 100 keys of random bytes:

Old: One loop took 298.645066ms with errCnt 22
New: One loop took 270.700415ms with errCnt 0

So current client takes longer per loop (one put request + one get request to the key from the put request). errCnt means how many times out of 100, it tries to talk to follower when the follower's log is not ready to answer the client request.

Looks good. Will wait on @mitake's fix for @jonboulle's feedback.

gyuho · 2015-12-23T17:47:59Z

@jonboulle Can you go over @mitake's updates to your feedback?

jonboulle · 2015-12-23T18:43:09Z

client/client.go

@@ -35,6 +35,7 @@ var (
 	ErrTooManyRedirects      = errors.New("client: too many redirects")
 	ErrClusterUnavailable    = errors.New("client: etcd cluster is unavailable or misconfigured")
 	errTooManyRedirectChecks = errors.New("client: too many redirect checks")


Nit, please swap lines 37/38

jonboulle · 2015-12-23T19:10:38Z

I still feel a bit uneasy about this for long-lived clients. Maybe I'm just being overly paranoid, but it still seems like there are various opportunities for things to get into a weird state where it's not really a PrioritizeLeader mode any more. For example, Do() code will just pin a new arbitrary (available) endpoint if anything goes wrong with a request.

xiang90 · 2015-12-23T19:13:10Z

@jonboulle Like I said, I will view this as an acceptable client side optimization when choosing the initial member to talk with without any guarantees. If user does not enable auto-sync, it probably means nothing after any configuration changes.

xiang90 · 2015-12-23T19:14:31Z

@jonboulle Another way is that etcd-server always piggy-back the leader addr as a hint. The client can always decide to switch to leader or not. That is what we did for etcd0 and some other systems do for reducing latency.

Current etcd client library chooses a default destination node from every member of a cluster in a random manner. However, requests of write and read (for consistent results) need to be forwarded to the leader node as the nature of Raft algorithm. If the chosen node is a follower, additional network traffic will be caused by the forwarding from follower to leader. Mainly for reducing the forward traffic, this commit adds a new mechanism for various endpoint selection mode to the client library which can be configured with client.Config.SelectionMode. Currently, two modes are provided: - EndpointSelectionRandom: default, same to existing behavior (pick a node in a random manner) - EndpointSelectionPrioritizeLeader: prioritize leader, for the above purpose I evaluated the effectiveness of the EndpointSelectionPrioritizeLeader with 4 t1.micro instances of AWS (3 nodes for etcd cluster and 1 node for etcd client). Client executes this simple benchmark (https://github.com/mitake/etcd-things/tree/master/prioritize-leader-bench), just writes 10000 keys. When SelectionMode == EndpointSelectionRandom (default), the benchmark needed 1 min and 32.102 sec to finish. When SelectionMode == EndpointSelectionPrioritizeLeader, the benchmark needed 1 min 4.760 sec.

mitake · 2015-12-24T02:03:52Z

@jonboulle fixed based on your comment, PTAL.

mitake · 2015-12-24T02:09:10Z

@jonboulle I understand your concern about the difference between actual server configuration and view from clients. However, such inconsistency can be caused by other reasons e.g. etcd cluster membership reconfiguration. Developers of long living clients must care about the inconsistency any way. I'd like to add this point to the docs.

jonboulle · 2015-12-24T11:54:48Z

OK, thanks for your patience.

Developers of long living clients must care about the inconsistency any way. I'd like to add this point to the docs.

This would be great, thanks

jonboulle · 2015-12-24T11:55:07Z

LGTM

client: add a mechanism for various endpoint selection mode

gyuho · 2015-12-24T14:54:04Z

Thanks Jon!

Sincerely,
Gyu-Ho Lee
On Dec 24, 2015 03:55, "Jonathan Boulle" notifications@github.com wrote:

Merged #4030 #4030.

—
Reply to this email directly or view it on GitHub
#4030 (comment).

mitake · 2015-12-24T15:30:28Z

@xiang90 @gyuho @jonboulle thanks a lot for detailed review!

This commit adds a document that provides tips of how to use the go client library. Currently it describes how to use the client.SelectionMode parameter that is added in etcd-io#4030.

mitake mentioned this pull request Dec 21, 2015

client: an option for prioritizing a leader node as a request destina… #3950

Closed

gyuho reviewed Dec 21, 2015
View reviewed changes

mitake force-pushed the endpoint-selection branch from 767491e to 6eee6f0 Compare December 22, 2015 02:35

xiang90 reviewed Dec 22, 2015
View reviewed changes

mitake force-pushed the endpoint-selection branch from 6eee6f0 to 53dabda Compare December 22, 2015 06:00

xiang90 assigned gyuho Dec 22, 2015

xiang90 added the area/clientv3 label Dec 22, 2015

mitake force-pushed the endpoint-selection branch from 53dabda to 0d01926 Compare December 22, 2015 06:07

mitake force-pushed the endpoint-selection branch from 0d01926 to 7616765 Compare December 22, 2015 08:52

jonboulle reviewed Dec 22, 2015
View reviewed changes

mitake force-pushed the endpoint-selection branch from 7616765 to 5d78417 Compare December 23, 2015 13:52

client: a new API for obtaining a leader node information

53be840

mitake force-pushed the endpoint-selection branch 2 times, most recently from c716ecb to aa52a42 Compare December 23, 2015 14:30

jonboulle reviewed Dec 23, 2015
View reviewed changes

mitake force-pushed the endpoint-selection branch from aa52a42 to a46ffc6 Compare December 24, 2015 02:03

jonboulle added a commit that referenced this pull request Dec 24, 2015

Merge pull request #4030 from mitake/endpoint-selection

dac56fa

client: add a mechanism for various endpoint selection mode

jonboulle merged commit dac56fa into etcd-io:master Dec 24, 2015

mitake deleted the endpoint-selection branch December 24, 2015 15:29

mitake mentioned this pull request Jan 19, 2016

Documentation: add a doc for using the go client library #4231

Merged

mitake mentioned this pull request Aug 24, 2023

Add Config.DisableProposalForwardingCallback for controlling message level proposal forwarding etcd-io/raft#88

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: add a mechanism for various endpoint selection mode #4030

client: add a mechanism for various endpoint selection mode #4030

mitake commented Dec 21, 2015

gyuho Dec 21, 2015

mitake commented Dec 22, 2015

gyuho commented Dec 22, 2015

xiang90 Dec 22, 2015

xiang90 commented Dec 22, 2015

mitake commented Dec 22, 2015

xiang90 commented Dec 22, 2015

gyuho commented Dec 22, 2015

xiang90 commented Dec 22, 2015

mitake commented Dec 22, 2015

mitake commented Dec 22, 2015

jonboulle Dec 22, 2015

gyuho commented Dec 22, 2015

gyuho commented Dec 23, 2015

jonboulle Dec 23, 2015

jonboulle commented Dec 23, 2015

xiang90 commented Dec 23, 2015

xiang90 commented Dec 23, 2015

mitake commented Dec 24, 2015

mitake commented Dec 24, 2015

jonboulle commented Dec 24, 2015

jonboulle commented Dec 24, 2015

gyuho commented Dec 24, 2015

mitake commented Dec 24, 2015

client: add a mechanism for various endpoint selection mode #4030

client: add a mechanism for various endpoint selection mode #4030

Conversation

mitake commented Dec 21, 2015

gyuho Dec 21, 2015

Choose a reason for hiding this comment

mitake commented Dec 22, 2015

gyuho commented Dec 22, 2015

xiang90 Dec 22, 2015

Choose a reason for hiding this comment

xiang90 commented Dec 22, 2015

mitake commented Dec 22, 2015

xiang90 commented Dec 22, 2015

gyuho commented Dec 22, 2015

xiang90 commented Dec 22, 2015

mitake commented Dec 22, 2015

mitake commented Dec 22, 2015

jonboulle Dec 22, 2015

Choose a reason for hiding this comment

gyuho commented Dec 22, 2015

gyuho commented Dec 23, 2015

jonboulle Dec 23, 2015

Choose a reason for hiding this comment

jonboulle commented Dec 23, 2015

xiang90 commented Dec 23, 2015

xiang90 commented Dec 23, 2015

mitake commented Dec 24, 2015

mitake commented Dec 24, 2015

jonboulle commented Dec 24, 2015

jonboulle commented Dec 24, 2015

gyuho commented Dec 24, 2015

mitake commented Dec 24, 2015