[DISCUSS] Task manager update API to allow changing a task's interval

[gmmorris]

There should be an update API in the task manager that allows to change certain properties within a task. Whenever changing the task's interval, it should if possible, re-adjust the runAt.

[mikecote]

Updated the description to something more explicit of what is needed from the task manager.

[elasticmachine]

Pinging @elastic/kibana-stack-services (Team:Stack Services)

[gmmorris]

Making this a Discuss Issue (for a while at least) as I feel this requires some debate.

Current State

So currently, if I'm understanding thing correctly, when an Alert is created/enabled we schedule a task without specifying a runAt field or interval field to Task Manager.
What this means is the the Alert is always run "immediately" after creation/enablement.

As part of the implementation of Alerting we provide a TaskTypeDefinition for TaskManager which gets called whenever an Alert needs to be evaluated.
In the TaskTypeDefinition we use the ability of tasks to override the runAt field as part of their run to use the interval (which is specified on the Alert) to calculate when the next run should take place.

The issue we're hitting is that as the Alert is run immediately after creation/enablement, resulting in the next runAt being calculated based on the interval as it was at the that time, it doesn't get a chance to recalculate the runAt until it gets run again. This means that if we update the alert at any time, including changing the interval, the changes won't take affect until it is rerun by Task Manager at some later point.

Issues To Think About

There are a couple of things that jump out at me here

1. This means that the use of interval in Alerting is different than the use of interval in regular tasks in Task Manager, but I can't see a good reason for this distinction. It means we have two places to maintain and keep in mind when thinking about intervals, and also means behaviour can diverge over time. For example, if we implement updating as part of this issue, then the semantics between updating a task vs. an alert could easily diverge.
   In fact, as the interval is stored differently in alerts, an Alerting task doesn't actually appear to have an interval at all from Task Manager's perspective, which harms maintainability already.
2. In relation to the semantic around updating an interval, there is a nuance we need to discuss: if an interval on a task/alert is updated, _do we recalculate the runAt as (scheduledAt / lastRun) + interval or updatedTime + interval? My instinct is that the interval should be calculated relative to the last run (or in lieu of a previous run, the scheduledAt time), but want o hear thoughts.

Suggested Plan Of Action

1. Even though it isn't actually addressing the cause for this Issue, we begin by adding a reschedule api in Task Manager which is used to reschedule an already scheduled task, which will support updating the interval and will calculate the correct runAt based on the semantics that we'll agree after discussing issue 2 above.

2. Then we move Alerting to use Task Manager's built in intervals (as per Convert alerts to use task manager intervals #46001) as this will reduce the dangers in issue 1 above.

3. Change Alerting's update api to use Task Manager's new reschedule api.

Note: Steps 2 & 3 will likely be done together in a single PR, as separating them might break existing update functionality in Alerting... but not 100% yet.

---

Any thoughts?