Adding a liveness check for consuming segments before queries are sent

[dang-stripe]

Is it possible to add a check for consuming segments to only serve queries when they are close to caught up? This is similar to the consumption based status checker but in steady state for rebalances, etc: #9244

[Jackie-Jiang]

One way I can think of is to integrate the consumption status check into the segment state transition from OFFLINE to CONSUMING. Basically do not mark the segment ready before the consumption catch up. It can slow down the state transition (if it takes very lone, it can even time out, but we can probably put a time limit for the wait), but maybe the correct approach, which can apply to both startup and rebalance

[jadami10]

I think this gets into classic routing problems. There's some threshold of unavailable segments or failed queries where you may want to go back to routing everywhere instead of failing because only M/N replicas are available.