[CI] CcrRollingUpgradeIT#testIndexFollowing failure

[martijnvg]

ERROR   3.80s | CcrRollingUpgradeIT.testIndexFollowing <<< FAILURES!
   > Throwable #1: org.elasticsearch.client.ResponseException: method [PUT], host [http://[::1]:43255], URI [/follower_index3/_ccr/follow?wait_for_active_shards=1], status line [HTTP/1.1 400 Bad Request]
   > {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[upgraded-node-follower-0][127.0.0.1:36653][indices:admin/xpack/ccr/put_follow]"}],"type":"illegal_argument_exception","reason":"the leader and follower index settings must be identical"},"status":400}
   > 	at __randomizedtesting.SeedInfo.seed([A114A4C994A3D7F3:2ECAB4031417AA50]:0)
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+release-tests/x-pack/qa/rolling-upgrade-multi-cluster/build/testrun/v7.1.0#follower#twoThirdsUpgradedTestRunner/J0/temp/org.elasticsearch.upgrades.CcrRollingUpgradeIT_A114A4C994A3D7F3-001
   > 	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260)
   > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238)
   > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
  2> NOTE: test params are: codec=Lucene80, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3ab677d8), locale=ar-AE, timezone=SystemV/CST6
   > 	at org.elasticsearch.upgrades.CcrRollingUpgradeIT.followIndex(CcrRollingUpgradeIT.java:134)
  2> NOTE: Linux 3.10.0-957.5.1.el7.x86_64 amd64/Oracle Corporation 1.8.0_202 (64-bit)/cpus=16,threads=1,free=424806384,total=514850816
  2> NOTE: All tests run in this JVM: [CcrRollingUpgradeIT]
   > 	at org.elasticsearch.upgrades.CcrRollingUpgradeIT.testIndexFollowing(CcrRollingUpgradeIT.java:70)
   > 	at java.lang.Thread.run(Thread.java:748)
Completed [1/1] in 6.26s, 2 tests, 1 error, 1 skipped <<< FAILURES!

This rolling upgrade test only exists in master, and hasn't yet been backported.
I'm not yet able to reproduce this failure. Somehow the leader / follower settings are not aligned.

Build url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/444/console

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

I locally changed the TransportResumeFollowAction#validate(...) method to include the index settings of both follow and leader indices in the exception message:

Throwable #1: org.elasticsearch.client.ResponseException: method [PUT], host [http://[::1]:40807], URI [/follower_index3/_ccr/follow?wait_for_active_shards=1], status line [HTTP/1.1 400 Bad Request]
   > {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[upgraded-node-follower-0][127.0.0.1:33471][indices:admin/xpack/ccr/put_follow]"}],"type":"illegal_argument_exception","reason":"the leader [{\"index.number_of_shards\":\"1\"}] and follower index [{\"index.number_of_shards\":\"1\",\"index.version.upgraded\":\"8000099\"}] settings must be identical"},"status":400}

The follower index has the index.version.upgrade index setting and the leader index doesn't have that index setting. I suspect that the follower index got this setting when ccr restored the leader index on an upgraded node. The fix is that TransportResumeFollowAction#validate(...) should also ignore any index.version* settings.