proxy: smart (or maybe just configurable) routing

[davidquarles]

@yichengq I just stumbled across this year-old exchange, which didn't seem to ever make it into a proper feature request.

My use case:

I currently have a three-node cluster with the active members split between 3 data centers. Because the proxy does a basic round-robin under the hood, our proxies are connecting across regions two-thirds of the time, and there is a very obvious performance hit when this happens:

app1 ~ # time curl --cacert $cacert --cert $cert --key $key https://$inter_region_tls_endpoint:4001/v2/keys/foo
{"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":28150814,"createdIndex":28150814}}

real    0m0.049s
user    0m0.019s
sys 0m0.000s
app2 ~ # time curl --cacert $cacert --cert $cert --key $key https://$cross_region_tls_endpoint:4001/v2/keys/foo
{"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":28150814,"createdIndex":28150814}}

real    0m0.247s
user    0m0.018s
sys 0m0.003s

Perhaps -- given some optional flag -- we can utilize an alternative "nearest" load-balancing algorithm? I'm imagining dynamic (sanely bounded) weights for each endpoint based on a moving average of response times but there may be other / better ways. Alternatively, I can hack something out with HAProxy. Thoughts?

[raoofm]

+1

[xiang90]

@heyitsanthony What do you think? shall we add hard priorities to endpoints?

[heyitsanthony]

@xiang90 sure, this is probably something we should handle in the grpc balancer

[xiang90]

@heyitsanthony I plan to copy what dns srv supports: https://en.wikipedia.org/wiki/SRV_record#Provisioning_for_high_service_availability