Description
This issue is similar to, but distinct from, #39284.
How to Reproduce
- Set up a 5.x cluster
- Create at least one Watch that runs periodically
- Upgrade the cluster to 6.7
- Reindex the
.watchesindex using the Kibana Upgrade assistant - Create a new periodic watch.
Expected behavior: The newly created Watch will be executed on schedule, as will existing watches.
Observed behavior: The newly created Watch will not run, although existing Watches will continue to run as expected.
Updates to existing Watches work normally, but deleting a Watch will cause warnings in the logs that the Watch cannot be found.
This can also be reproduced by switching the .watches alias to point to a new index that does not start with .watches, even if not using the Upgrade Assistant.
Cause
When Kibana reindexes the .watches-6 index, it is reindexed to <INDEX_NAME>. Watcher sets up a listener on indices which have names that start with .watches to pick up newly created or deleted Watches.
Solution
Instead of using a prefix, Watcher should listen on whichever index is pointed to by the .watches alias.
Workaround
After adding a Watch, stopping and starting Watcher will cause the new Watch to be executed as expected, although this must be done each time Watches are added.