Enrich indices can be deleted if cluster leader fails over during policy execution

[jbaiera]

Enrich policy execution can take a significant amount of time due to the reindexing step. Because of this, enrich policies are limited to only have one execution at a time per policy. This is managed by a set of locks on the master node.

Enrich indices are cleaned up during a background process to avoid search shards disappearing randomly during ingest. This background maintenance process runs on the master node so that it can reference the lock states for all policies. This allows a long running policy to execute without the fear of the maintenance task deleting the index before it can be created.

There exists an edge case such that a policy that takes sufficiently long to execute can lead to all enrich indices for a policy to be deleted in the event that the acting master fails over.

Assuming a cluster with an enrich policy my_policy which has been previously executed, and thus has an enrich index .enrich-my_policy-0000000000001. The index holds an alias named .enrich-my_policy which demarcates it as the "current" enrich index for the policy.

Scenario:

1. Execute my_policy enrich policy.
2. Master node obtains the my_policy lock and submits the request to a data node to perform.
3. Data node creates new index .enrich-my_policy-0000000000002 and configures the mapping with the required meta fields.
4. The data node begins reindexing data into the new index. This process must take longer than 15 minutes for this bug to surface.
5. The master node is shutdown suddenly. All enrich lock state is lost.
6. The policy execution request likely times out/fails for the client, but the policy is still executing on the data node.
7. A new master node takes control of the cluster.
8. Fifteen minutes after becoming the cluster leader, the master node executes the enrich maintenance task. The task locates .enrich-my_policy-0000000000002. The task sees that there are no locks indicating that the execution is in progress for this and assumes the index is abandoned. The maintenance task removes the index.
9. The policy that is still executing the reindex operation continues to submit bulk indexing requests. The bulk index operation automatically recreates .enrich-my_policy-0000000000002, but without the metadata in the index mapping that identifies it as an enrich index.
10. The policy execution completes and the interloper enrich index is promoted to the "current" one by setting its alias.
11. Some time later, the maintenance task executes again. This time, it removes the previous valid enrich index .enrich-my_policy-0000000000001 because it does not hold the "current" alias for the policy. The maintenance task also removes the .enrich-my_policy-0000000000002 index again, because even though it has the current alias, it does not have the metadata in its mappings to identify it as an enrich index.
12. At this point, the my_policy enrichment policy is left with no valid enrich indices.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[jbaiera]

We discussed this in the Data Management area sync today and came to a few conclusions:

1. We should update the policy execution logic to check the mapping metadata one last time before promoting an index. This would at least help guard from this problem happening. It does not fix the problem of the locks disappearing during a master restart, nor does it fully fix each race condition since the final promotion step may be running concurrently with the cleaner on another master node.
2. Enrich policy executions should be modified to be cancellable. We have the ability to detect when the client submitting a request disappears. We should see if we can leverage that to cancel the policy execution in the event that the master node that submitted it has disappeared.
3. When enrich was initially developed, the locks were colocated in the same JVM as the policy execution logic. As the feature grew, the policy execution logic was delegated off of the master node and run on another machine. The local enrich locks on the active master have been a liability for this kind of problem ever since.
4. We want to eventually revisit how enrich policies get executed. One possible option is to store the currently executing policies in the cluster state. This has some implications that need to be addressed regarding their cleanup in case of errors.
5. Another option we have is to create an affinity for the policy executing node based on the index it is creating. Instead of submitting the enrich execution to a random node, we could create the target index first, then submit the execution to whichever node holds shard 0's primary. This makes an ongoing policy execution easier to locate in case of coordinator failure.
6. Options 4 and 5 above will require some design work before any changes can be made to enrich features.

[jbaiera]

4. We want to eventually revisit how enrich policies get executed. One possible option is to store the currently executing policies in the cluster state. This has some implications that need to be addressed regarding their cleanup in case of errors.
5. Another option we have is to create an affinity for the policy executing node based on the index it is creating. Instead of submitting the enrich execution to a random node, we could create the target index first, then submit the execution to whichever node holds shard 0's primary. This makes an ongoing policy execution easier to locate in case of coordinator failure.
6. Options 4 and 5 above will require some design work before any changes can be made to enrich features.

Since the suggested options are talking about either placing executing enrich policies in the cluster state or moving policy executions to a node based on some kind of affinity, I wonder if it makes more sense to simply convert enrich policy executions into a persistent task. The execution will be submitted to a non-master node, it can be looked up from the cluster state, we can detect from the executor if a policy execution is already in progress when the task is allocated. This might be worth exploring instead.