Prevent import of dangling indices from a later version

[Bukhtawar]

A higher version of an index 6.3.2 can be restored on a cluster with node version 6.3.1 however when additional nodes try to join the cluster, their membership is restricted based on compatibility checks
which ensures that all indices in the given metadata will not be created with a newer version of elasticsearch as well as that all indices are newer or equal to the minimum index compatibility version.
based on Version#minimumIndexCompatibilityVersion

Elasticsearch version (bin/elasticsearch --version): 6.3.1

Plugins installed: []

JVM version (java -version): JDK 10

OS version (uname -a if on a Unix-like system): Linux ip-10-212-18-25 4.9.70-25.242.amzn1.x86_64

Description of the problem including expected versus actual behavior:

Steps to reproduce:

1. Restore an index of version 6.3.2 on a datanode with version 6.3.1
2. Spin up a new data node with version 6.3.1
3. Notice node is unable to join the cluster

Provide logs (if relevant):
[2018-10-03T00:00:10,030][INFO ][o.e.d.z.ZenDiscovery ] [wowvdMi] failed to send join request to master [{5GZT6AM}{5GZT6AMsSSy2x5vCdDLFXA}{1aYFoazmRXWt2OKoo3Slfg}{10.xx.xxx.xxx}{10.xx.xxx.xxx:9300}], reason [RemoteTransportException[[5GZT6AM][10.xx.xx.xxx:9300][internal:discovery/zen/join]]; nested: IllegalStateException[index [index-docs_2018092113/fKi49vf3TzKDshgg9ydzaQ] version not supported: 6.3.2 the node version is: 6.3.1]; ]

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

I have this reproduced => fixing now to not allow restoring a newer index version to an older datanode version.

[original-brownbear]

I talked about this with @DaveCTurner and the behavior here seems to not necessarily be a bug.

We are not actually restoring any data to the old data node in a mixed cluster, all we're doing is acknowledging the restore which puts the newer index in the state and prevent old datanodes from joining.

With the change I suggested in #34676 we're preventing putting starting the restore during rolling upgrade which may be less confusing but also reduces functionality during rolling upgrade (which seems to work fine if you have new version data nodes present).

=> maybe the behavior is ok as is?

[DaveCTurner]

I'd like more information from @Bukhtawar because I tried the steps given to reproduce this and got the expected error when trying to restore a snapshot taken on a 6.4.2 cluster into a 6.4.1 cluster:

POST /_snapshot/my_backup/snapshot_1/_restore
# 500 Internal Server Error
# {
#   "status": 500,
#   "error": {
#     "reason": "[my_backup:snapshot_1/qHahtNWXQ3elpe8rYqHEkA] the snapshot was created with Elasticsearch version [6.4.2] which is higher than the version of this node [6.4.1]",
#     "root_cause": [
#       {
#         "reason": "[my_backup:snapshot_1/qHahtNWXQ3elpe8rYqHEkA] the snapshot was created with Elasticsearch version [6.4.2] which is higher than the version of this node [6.4.1]",
#         "type": "snapshot_restore_exception"
#       }
#     ],
#     "type": "snapshot_restore_exception"
#   }
# }

To reproduce this needed a cluster that comprised a mix of 6.4.1 and 6.4.2 version nodes. We only expect a cluster of mixed versions to occur during a rolling upgrade, and in this situation we don't expect there to be more 6.4.1 nodes joining the cluster - in fact there are other things you can do during a rolling upgrade that will block the older nodes from joining the cluster, such as simply creating an index, and the solution in all cases is to upgrade the removed node. I think perhaps the docs could be clearer on this subject, but I can't see that we should change the behaviour here.

Did this occur during a rolling uprgade, and if so why were there more 6.4.1 nodes joining the cluster?