Add acquire shard time metric #2912
Conversation
service/history/shardContext.go
Outdated
| @@ -127,6 +129,7 @@ type ( | |||
|
|
|||
| // exist only in memory | |||
| standbyClusterCurrentTime map[string]time.Time | |||
| shardOwnershipChanged bool | |||
There was a problem hiding this comment.
ownershipChanged is a bit confusing to me - (1) does it mean this shard is no longer valid because ownership has moved to another host or (2) does it mean this shard is loaded because of a previous ownership change. If I understand right, you want (2) ? Is that right ? If that's the case, how do you feel about storing "previousOwner" here instead of this flag.
There was a problem hiding this comment.
This flag means that while we were trying to acquire a shard, record that we've loaded from the database had a different owner than the current host's identity. Hence it means that shard ownership has changed and this is the first time that we've discovered it.
| if context.ShardOwnershipChanged() { | ||
| i.GetMetricsClient().RecordTimer(metrics.ShardInfoScope, metrics.ShardItemAcquisitionLatency, | ||
| context.GetCurrentTime(i.GetClusterMetadata().GetCurrentClusterName()).Sub(context.GetLastUpdatedTime())) | ||
| } |
There was a problem hiding this comment.
I am not sure if this can give you accurate value for shard_ownership_change time - UpdateShard is called only every minute unless we run out of rangeID space, in which case, it is immediately invoked.
There was a problem hiding this comment.
Yep this is an approximation, however it would provide an upper boundary for the time when shard is not owned by anyone, I plan to also add code to ensure that shards are updated during shutdown to ensure most up to date timestamps.
Record a metric that measures time since last shard update each time when ownership changes.