Task manager can get in an endless loop when attempting to claim non-saved object tasks

[mikecote]

It seems possible when users upgrade to 7.4 that an old version of Kibana re-inserts tasks within the .kibana_task_manager index without the id prefix of task:. When this scenario happens, task manager would get in an infinite loop attempting to claim those tasks but doing the update call on the object containing the prefix (task:).

For example, a task with id of oss_telemetry-vis_telemetry would cause task manager to attempt claiming a task with an id of task:oss_telemetry-vis_telemetry. This would cause a 409 to be returned since the version returned on read doesn't match the document version of the other.

In 7.4, it would cause for example 200 update requests per second of attempting to claim a task while constantly getting a 409 as a result. In 7.5 the issue isn't as severe but every 30 seconds a log like this shows up: [error][task_manager] Failed to mark Task vis_telemetry "oss_telemetry-vis_telemetry" as running: Task has been claimed by another Kibana service.

We should make sure when searching / claiming tasks that we also make sure they're saved objects. Similar to how the migrator only migrates documents that are saved objects: https://github.com/elastic/kibana/blob/master/src/core/server/saved_objects/migrations/core/migrate_raw_docs.ts#L41.

Could be similar to a 7.1 issue? #47607

[elasticmachine]

Pinging @elastic/kibana-stack-services (Team:Stack Services)

[ash9146]

Same Error Kibana 7.6 version
7.4 > 7.6 Upgrade ...

Is there a fundamental solution? / How to solve?
Other than the log, it seems to be working normally.

[mikecote]

@ash9146 to solve the issue, the objects without task: prefix within the .kibana_task_manager index need to be removed.

[ash9146]

@mikecote Thanks. I solved

1. List up, not prefix 'task:'
   GET /.kibana_task_manager_26/_search
   
   

2. Remove document
   DELETE /.kibana_task_manager_26/_doc/oss_telemetry-vis_telemetry
DELETE /.kibana_task_manager_26/_doc/Maps-maps_telemetry

3. Logging verification
   Good!

[wixaw]

Hello , this is not resolve the problem for me
I have delete all DELETE /.kibana_task_manager_*
but this is recreate everytime
{"type":"log","@timestamp":"2020-05-29T14:21:18Z","tags":["error","plugins","taskManager","taskManager"],"pid":26748,"message":"Failed to mark Task vis_telemetry \"oss_telemetry-vis_telemetry\" as running: Task has been claimed by another Kibana service"}

ElasticStack version : 7.7.0

[gmmorris]

Hi @wixaw
Sorry you've encountered this issue, lets see if we can figure out what's wrong.
Generally speaking this error would mean that you have two (or more) Kibana instances and they're having trouble resolving an owner for the task and completing it and usually wiping the kibana_task_manager_* indicies is enough unless something recreates the task incorrectly.

Could you expand a bit about your setup?
It would help to know how many instances of Kibana you're running. A single instance or multiple?
Is this error causing Kibana to crash in any way, or is it simply repeatedly appearing in your logs?
Does the error reappear spontaneously or after a restart of Kibana?

Are there any other errors in the logs that might give a hint to what's going on?

[wixaw]

Hello @gmmorris
I only have one kibana instance, Maybe a second could have started at one point but it is stopped now.

This is only in the logs, I wanted to discover the free alert system in the kibana logs and if we have to manage events in this one, it is rather a problem to have these false alerts.

This alert appears approximately every 15 seconds without ever stopping.

Thank you

[wixaw]

Hello @gmmorris
I have same issue on 7.9.0
Thanks