extraneous error/warning messages when deleting alerts

[pmuellr]

Kibana version: master

Steps to reproduce:

start the functional test server, then run the following commands at a terminal

$ npm install -g pmuellr/kbn-action
...

$ export KBN_URLBASE=http://elastic:changeme@localhost:5620

$ kbn-action create .server-log server-log {} {}
{
    "id": "ec1783c9-38ae-4fac-91d3-ad1fd78e2faa"
}

$ export ACTION_ID=ec1783c9-38ae-4fac-91d3-ad1fd78e2faa

$ kbn-alert create test.always-firing 5s '{index:test_always_firing}' \
  "[{group:default id:'$ACTION_ID' params:{message: 'from alert 5s'}}]"
{
    "id": "c832e950-b494-11e9-b723-f594131b7bfc",
    ...
}

$ { run same command as above, creating another alert }
{
    "id": "c4475740-b494-11e9-b723-f594131b7bfc",
    ...
}

...

$ kbn-alert delete c832e950-b494-11e9-b723-f594131b7bfc
kbn-alert.js: status code 204
body: ""

$ { delete more of the alerts, based on id }
kbn-alert.js: status code 204
body: ""

After deleting alerts, you may see one or both of these messages in the Kibana log:

[error][task_manager] Task alerting:test.always-firing
    "c682c850-b494-11e9-b723-f594131b7bfc"
    failed: Error: Saved object [alert/c6205d00-b494-11e9-b723-f594131b7bfc] not found

[warning][task_manager] Task alerting:test.always-firing
    "a4f9e1f0-b494-11e9-b723-f594131b7bfc"
    failed in attempt to run:
    Saved object [task/a4f9e1f0-b494-11e9-b723-f594131b7bfc] not found

Expected behavior:

Should not display error/warning messages when firing alerts that have recently been deleted. Or at least should be fairly rare, and not scary (warning not error), in case it's difficult to precisely nail this down.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-stack-services

[mikecote]

Solution should also handle the scenario where a space gets deleted and the task becomes orphaned. The task currently retries at the regular interval but should stop retrying and not log error messages.

[pmuellr]

Assuming we can detect the case of "this alert wants to run but it's space no longer exists", I think we should mark these alerts as disabled, and then leave them there. They'll just be taking up space in the SO at that point, but won't cause queueing up in TM.

We should have some kind of UI somewhere that we can see these, so a customer can clean them up, attempt to move them to another space, etc. Probably needs to be separate from "show me the disabled alerts", since that would mix customer-disabled alerts with these orphans.

Guessing we may have other cases of "orphaned" alerts - eg, alertType not registered. Maybe we need a special way to note these disabledBySystem: boolean, disabledBySystemReason: string kinda thing.

[mikecote]

Assuming we can detect the case of "this alert wants to run but it's space no longer exists", I think we should mark these alerts as disabled

What happens when a space gets deleted is all the saved objects within the space also get deleted. But since tasks are space agnostic, only the alert gets deleted and not the task. This causes 404 errors when the task runs and attempts to find the related alert.

[pmuellr]

Ah, in that case it's certainly worth a log message, here's the message we're seeing (I guess):

[error][task_manager] Task alerting:test.always-firing
    "c682c850-b494-11e9-b723-f594131b7bfc"
    failed: Error: Saved object [alert/c6205d00-b494-11e9-b723-f594131b7bfc] not found

It would be great if we could provide the spaceId for this SO as well as the id, would give customers a hint at why this is being logged. Saved object [alert/abc] not found in space blort kinda thing.

[mikecote]

Looks like this issue is becoming more common. With the new error handling introduced in #53688, the alert task will keep getting scheduled indefinitely. A probably scenario for this to happen is if Kibana restarts / crashes between the remove of the alert saved object and the removal of the task object. There may be others..

Worth fixing / backporting to 7.7 if it only requires a few lines of code.

[gmmorris]

With the new error handling introduced in #53688, the alert task will keep getting scheduled indefinitely

I don't think this is true - the error is thrown because the task was in flight when it was removed from the store and then when the task completed it failed to save as the task no longer exists.
I tested locally and it doesn't look like it reschedules - as the task is no longer in the store to be rescheduled.

I'm looking into whether there's a graceful way to detect that the task is deleted and not show an error, but I'm afraid of accidentally swallowing up real errors that should be logged, so not sure it's the right approach. To be seen.