Reallocation of .watches shards resets interval schedule watches

[masseyke]

If I create a watch on a 3-node cluster with an interval schedule, like

curl -X PUT "localhost:9200/_watcher/watch/log_error_watch?pretty" -H 'Content-Type: application/json' -d'
{
  "trigger" : {
    "schedule" : { "interval" : "1h" } 
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logs" ],
        "body" : {
          "query" : {
            "match" : { "message": "error" }
          }
        }
      }
    }
  }
}
'

And then I move a .watches shard:

curl -X POST "localhost:9200/_cluster/reroute?metric=none&pretty" -H 'Content-Type: application/json' -d'
{
  "commands": [
    {                      
      "move": {
        "index": ".watches", "shard": 0,
        "from_node": "runTask-2", "to_node": "runTask-1"
      }
    }
  ]
}
'

Then the trigger interval starts over -- it will be one hour from the time I moved the shard. This especially becomes a problem if the interval is large because shards are more likely to get moved around by the allocator in more recent versions of Elasticsearch. For example if a 4-hour interval is configured, and .watches shards get moved on average a couple of times per hour, it could be days before the watch is ever executed.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)