[Uptime] Add 'Stale' monitor state

[andrewvc]

Describe the feature:

Currently on the overview page we show monitors in one of 2 states. Up, or down. In heartbeat 7.6+ we now have the ability to compute a third 'dead' state, where the time range includes a monitor, but the monitor itself has missed its next expected run.

Consider the timeline below, where tC indicates a check has been sent, and the letter | indicates the time the next expected check is supposed to happen.

Monitor A:------C-----|C-------|C-------|C>------|
Monitor B:------C-----|C-------|---------------------
------------------- time --------------------> (now)

Monitor A in this example is alive, it's next expected check is soon, but it hasn't missed its schedule in any significant way. Monitor B sent two checks, but never sent the third expected check. We can safely mark it as dead.

This issue proposes that we

1. Add a third category of 'dead' monitors. (Defined as monitors that have not received an expected check in > 5m by default).
2. By default do not display dead monitors, with a toggle switch on the overview page making them optionally visible (and a count indicating the number of dead monitors).
3. Do not show the last status of dead monitors (for perf reasons)

Describe a specific use case for the feature:

Using the app today, if a user deletes a monitor ID it stays around when looking at longer history ranges. This can create quite a bit of clutter and be confusing. Additionally, the performance optimization introduced in #52433 has some odd corner cases where the new optimal strategy doesn't return status data for dead monitors. Rather than 'fix' that behavior we could more clearly indicate the actual state of these monitors.

@katrin-freihofner would esp. appreciate your feedback here.
@katrin-freihofner @drewpost @justinkambic @shahzad31

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[shahzad31]

I agree with the approach we do need it regardless of the the need imposed by corner cases caused by performance optimization.

We can discuss naming though, should we call them dead monitors/Inactive. On UI, having toggle filter should be fine, i will suggest another thing, once Dead monitor shows up in list, we can color mark them by reducing opacity of that monitor row.

As far as threshold goes, i feel like we should give the user choice , we can make the default threshold of 5 mins. But in UI in front of that toggle, we can have some sort of select filter, saying show dead monitors for 5, 10, 20, 30 mins etc . Though we can also configure it via yaml file.

I do wonder how much the optimization has cost us , seems like lots of effort to maintain it :)

[andrewvc]

@shahzad31 what do you think about stale instead? That was proposed by @drewpost in a conversation we had.

I agree with making it a choice for the amount of time. We can add that setting to the new settings page I'm working on in #53550

I think that this optimization is actually leading us to a better UX. So, while it may appear as a cost here, it's making us ask why we're showing the status of something that has a stale status. So I'd call this a UX enhancement. The status is not up/down for something that missed a check, it's actually stale, it's got an unknown state.

[andrewvc]

One more note about the optimization here, it's the difference between 20+ second long queries for large data sets vs. 3-5s queries, so I think we need it.

[shahzad31]

Will it make significant difference once we remove those scripts from query?

[andrewvc]

@shahzad31 If you mean the scripted metric query, that has a negligible performance impact (maybe a constant 200ms, which isn't great, but isn't that bad either).

[katrin-freihofner]

Do you mean something like this?

[shahzad31]

Can't we just add it to Filtergroup?