[CI] CCR: testFollowIndexAndCloseNode fails

[dnhatn]

testFollowIndexAndCloseNode fails on 6.x:

ERROR   58.1s J1 | ShardChangesIT.testFollowIndexAndCloseNode <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <0>
   >      but: was <3>
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.xpack.ccr.ShardChangesIT.lambda$unfollowIndex$12(ShardChangesIT.java:533)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:847)
   > 	at org.elasticsearch.xpack.ccr.ShardChangesIT.unfollowIndex(ShardChangesIT.java:519)
   > 	at org.elasticsearch.xpack.ccr.ShardChangesIT.testFollowIndexAndCloseNode(ShardChangesIT.java:351)
   > 	at java.lang.Thread.run(Thread.java:748)

This may be the actual reason.

  2> WARNING: Uncaught exception in thread: Thread[elasticsearch[node_td4][write][T#1],5,TGRP-ShardChangesIT]
  2> java.lang.AssertionError: seqNo [133] was processed twice in generation [2], with different data. prvOp [Index{id='CBdenGUBUjHVaMehll6c', type='doc', seqNo=133, primaryTerm=1, version=1, autoGeneratedIdTimestamp=-1}], newOp [Index{id='CBdenGUBUjHVaMehll6c', type='doc', seqNo=133, primaryTerm=2, version=1, autoGeneratedIdTimestamp=-1}]
  1> [2018-09-02T17:19:26,895][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_tm2] removing template [random_index_template]
  2> 	at __randomizedtesting.SeedInfo.seed([26198A03C82BA27]:0)
  1> [2018-09-02T17:19:26,903][INFO ][o.e.n.Node               ] [testValidateFollowingIndexSettings] stopping ...
  2> 	at org.elasticsearch.index.translog.TranslogWriter.assertNoSeqNumberConflict(TranslogWriter.java:214)
  2> 	at org.elasticsearch.index.translog.TranslogWriter.add(TranslogWriter.java:181)
  1> [2018-09-02T17:19:26,906][INFO ][o.e.c.s.MasterService    ] [node_tm2] zen-disco-node-left({node_tc4}{mIMSWWHESX-PxSM6NVVikw}{ZqmDtVfPSTaqXDkLLas5WA}{127.0.0.1}{127.0.0.1:33427}{xpack.installed=true}), reason(left), reason: removed {{node_tc4}{mIMSWWHESX-PxSM6NVVikw}{ZqmDtVfPSTaqXDkLLas5WA}{127.0.0.1}{127.0.0.1:33427}{xpack.installed=true},}

CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/2388/console
Log: testFollowIndexAndCloseNode.txt.zip

[elasticmachine]

Pinging @elastic/es-distributed

[dnhatn]

I've muted this test on master and 6.x.

[dnhatn]

This test failed because we assigned different terms (1 and 2) to seq=133. This happened in these steps:

1. The primary of the follower assigns its primary term to operations in TransportBulkShardOperationsAction.

2. An operation (seq=133) is assigned term=1, indexed to the primary and the replica of the follower. However, this primary is restarted before it can respond to ShardFollowNodeTask.

3. The replica is promoted, and the old primary recovers operation {seq=133, term=1} from the new primary (i.e. old replica) via peer-recovery.

4. ShardFollowNodeTask does not receive a response, then retries to index the operation(seq=133) to a new primary. The new primary assigns its term (term=2) to operation (seq=133). TransportBulkShardOperationsAction replicates this operation {seq=133,term=2} to the old primary.

5. The new operation {seq=133,term=2} conflicts with the old operation {seq=133,term=1} and trip the assertion.

java.lang.AssertionError: seqNo [133] was processed twice in generation [2], 
with different data. 

prvOp [Index{id='CBdenGUBUjHVaMehll6c', type='doc', seqNo=133, primaryTerm=1,
             version=1, autoGeneratedIdTimestamp=-1}], 
newOp [Index{id='CBdenGUBUjHVaMehll6c', type='doc', seqNo=133, primaryTerm=2,
             version=1, autoGeneratedIdTimestamp=-1}]

One solution I can see is to let ShardFollowNodeTask assign the primary term once before dispatching operations. We can piggyback the current term on the primary in responses (even though we won't have the latest term on ShardFollowNodeTask all the time). @bleskes and @jasontedor WDYT?

[dnhatn]

This still is an issue with the proposal if ShardFollowNodeTask is restarted. Or let's relax the assertNoSeqNumberConflictassertion for the following engines?

[bleskes]

I think this specific situation will be avoided with rollbacks in place, correct? I did make me think of a deeper problem. I added this as a discussion topic for our next sync.

[dnhatn]

I think this specific situation will be avoided with rollbacks in place, correct?

No, as we are testing with 1 replica on both sides.

[dnhatn]

I've un-muted this test. It should be okay now as the FollowingEngine will skip an operation which was already processed before (see #34099).

I am closing this issue since @jasontedor is working on a broader fix which covers this issue.

/cc @martijnvg

[dnhatn]

We haven't resolved this issue completely. I will mute this test.
CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/122/console