add exp backoff to grpc client reconnects to vals

[andymck]

Adds exponential backoff to reconnects to validators. The reconnects will start at 5 sec intervals and with each attempt expodentially increase up to a max of 180 secs. Reconnects include jitter

[Vagabond]

Shouldn't we increase the backoff if the MAX_BLOCK_AGE is too high, as well?

[andymck]

Shouldn't we increase the backoff if the MAX_BLOCK_AGE is too high, as well?

if the max block age is exceeded, the statem will reenter the setup state and thus we will hit the backoff

[Vagabond]

Shouldn't we increase the backoff if the MAX_BLOCK_AGE is too high, as well?

if the max block age is exceeded, the statem will reenter the setup state and thus we will hit the backoff

ok, this is what I figured.