Cortex 1.0.0 ingesters rollout breaks zone set in the ring

[pracucci]

While upgrading to Cortex 1.0.0 and doing a rolling update of ingesters, I've noticed only the last rolled out ingester has a zone set in the ring, while all previous one (already rolled out) haven't.

The issue is caused by the co-existence of old and new ingesters. Old ingesters don't know the zone in the ring data structure so when they do deserialise the ring, the zone is lost. This issue disappear once the last ingester rollout is completed but at this point the zone for all ingesters (except the last one) is already lost.

[pracucci]

A manual hack to fix it is forgetting all ingesters without a zone once the rolling update is completed.

[khaines]

Thanks, @pracucci! I'll do some reproduction of this and think about a solution.

[khaines]

I think I see the bug here : https://github.com/cortexproject/cortex/blob/master/pkg/ring/lifecycler.go#L667

This is the block that the ingesters (or other ring clients) use to update their ring record on the heartbeat timer. It writes out address data but not the zone, so if a pre-1.0 overwrote a blank entry here, the ingester would not correct it on the following heartbeat cycle. The manual fix of forgetting the ingester causes the other part of this if block to run and re-registers the ingester (which sets the zone).

I'll be focusing on this area for the fix and test(s).