[CI] testRetentionLeasesSyncOnRecovery fails

[hendrikmuhs]

IMPORTANT : This test is still muted on master and failure applies to several versions.

Relates to #38764 - but I think this is a new issue.

Old fix: 5fad38e

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+periodic/77/console

The test compares 2 maps, which are identical by content but differently ordered.

Log:

java.lang.AssertionError: 
Expected: <{SaqDQEfE=RetentionLease{id='SaqDQEfE', retainingSequenceNumber=4119153756517795675, timestamp=1550569432369, source='vbgznlgo'}, eFvarfWn=RetentionLease{id='eFvarfWn', retainingSequenceNumber=5153974965419564929, timestamp=1550569432169, source='pcjCyKmZ'}, ycwCgVUO=RetentionLease{id='ycwCgVUO', retainingSequenceNumber=1537622719951565698, timestamp=1550569432169, source='mWZziHld'}, YzHeeQlt=RetentionLease{id='YzHeeQlt', retainingSequenceNumber=6253389575976647540, timestamp=1550569432169, source='lHsrBCLa'}, NZMLgOsO=RetentionLease{id='NZMLgOsO', retainingSequenceNumber=9123738011503356278, timestamp=1550569432169, source='hIqIRyLV'}}>
     but: was <{YzHeeQlt=RetentionLease{id='YzHeeQlt', retainingSequenceNumber=6253389575976647540, timestamp=1550569432169, source='lHsrBCLa'}, NZMLgOsO=RetentionLease{id='NZMLgOsO', retainingSequenceNumber=9123738011503356278, timestamp=1550569432169, source='hIqIRyLV'}, eFvarfWn=RetentionLease{id='eFvarfWn', retainingSequenceNumber=5153974965419564929, timestamp=1550569432169, source='pcjCyKmZ'}, ycwCgVUO=RetentionLease{id='ycwCgVUO', retainingSequenceNumber=1537622719951565698, timestamp=1550569432169, source='mWZziHld'}, SaqDQEfE=RetentionLease{id='SaqDQEfE', retainingSequenceNumber=4119153756517795675, timestamp=1550569432169, source='vbgznlgo'}}>
	at __randomizedtesting.SeedInfo.seed([D9CD8E08DAD8675F:93D9EB615232A8A8]:0)

/CC @jasontedor

[elasticmachine]

Pinging @elastic/es-distributed

[jasontedor]

Note the test that is failing here is testRetentionLeasesSyncOnRecovery, not testRetentionLeasesSyncOnRemove.

04:44:03  FAILURE 2.10s J1 | RetentionLeaseIT.testRetentionLeasesSyncOnRecovery <<< FAILURES!
04:44:03     > Throwable #1: java.lang.AssertionError: 
04:44:03     > Expected: <{SaqDQEfE=RetentionLease{id='SaqDQEfE', retainingSequenceNumber=4119153756517795675, timestamp=1550569432369, source='vbgznlgo'}, eFvarfWn=RetentionLease{id='eFvarfWn', retainingSequenceNumber=5153974965419564929, timestamp=1550569432169, source='pcjCyKmZ'}, ycwCgVUO=RetentionLease{id='ycwCgVUO', retainingSequenceNumber=1537622719951565698, timestamp=1550569432169, source='mWZziHld'}, YzHeeQlt=RetentionLease{id='YzHeeQlt', retainingSequenceNumber=6253389575976647540, timestamp=1550569432169, source='lHsrBCLa'}, NZMLgOsO=RetentionLease{id='NZMLgOsO', retainingSequenceNumber=9123738011503356278, timestamp=1550569432169, source='hIqIRyLV'}}>
04:44:03     >      but: was <{YzHeeQlt=RetentionLease{id='YzHeeQlt', retainingSequenceNumber=6253389575976647540, timestamp=1550569432169, source='lHsrBCLa'}, NZMLgOsO=RetentionLease{id='NZMLgOsO', retainingSequenceNumber=9123738011503356278, timestamp=1550569432169, source='hIqIRyLV'}, eFvarfWn=RetentionLease{id='eFvarfWn', retainingSequenceNumber=5153974965419564929, timestamp=1550569432169, source='pcjCyKmZ'}, ycwCgVUO=RetentionLease{id='ycwCgVUO', retainingSequenceNumber=1537622719951565698, timestamp=1550569432169, source='mWZziHld'}, SaqDQEfE=RetentionLease{id='SaqDQEfE', retainingSequenceNumber=4119153756517795675, timestamp=1550569432169, source='vbgznlgo'}}>
04:44:03     > 	at __randomizedtesting.SeedInfo.seed([D9CD8E08DAD8675F:93D9EB615232A8A8]:0)
04:44:03     > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
04:44:03     > 	at org.elasticsearch.index.seqno.RetentionLeaseIT.testRetentionLeasesSyncOnRecovery(RetentionLeaseIT.java:370)
04:44:03     > 	at java.lang.Thread.run(Thread.java:748)

[jasontedor]

Also, the maps are not equal, there is a timestamp difference.

Expected: <{SaqDQEfE=RetentionLease{id='SaqDQEfE', retainingSequenceNumber=4119153756517795675, timestamp=1550569432369, source='vbgznlgo'}

but: was..., SaqDQEfE=RetentionLease{id='SaqDQEfE', retainingSequenceNumber=4119153756517795675, timestamp=1550569432169, source='vbgznlgo'}}>

[jasontedor]

I found the issue, test bug. I will push a fix shortly.

[romseygeek]

This test has failed again on 6.7: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+internalClusterTest/2792/console

java.lang.AssertionError: 
Expected: <{XGXiyXac=RetentionLease{id='XGXiyXac', retainingSequenceNumber=228887311162678927, timestamp=1552991928992, source='OWVEeRER'}, UfXemsvH=RetentionLease{id='UfXemsvH', retainingSequenceNumber=6460015575321227402, timestamp=1552991928992, source='RCaDMWcA'}, yxZjoXOD=RetentionLease{id='yxZjoXOD', retainingSequenceNumber=6274212870258453507, timestamp=1552991928992, source='QeMpImcA'}, VyiIjJYs=RetentionLease{id='VyiIjJYs', retainingSequenceNumber=2133028230013912907, timestamp=1552991928992, source='yQsjFtqy'}, sfjfrVor=RetentionLease{id='sfjfrVor', retainingSequenceNumber=8701391374091472416, timestamp=1552991928992, source='uFWFHpJw'}, pYQTNwQT=RetentionLease{id='pYQTNwQT', retainingSequenceNumber=4919630631661594402, timestamp=1552991928992, source='hdrfFeBC'}, oWrKJWSr=RetentionLease{id='oWrKJWSr', retainingSequenceNumber=7602602061015029303, timestamp=1552991929193, source='ZvAAwqpt'}}>
     but: was <{XGXiyXac=RetentionLease{id='XGXiyXac', retainingSequenceNumber=228887311162678927, timestamp=1552991928992, source='OWVEeRER'}, UfXemsvH=RetentionLease{id='UfXemsvH', retainingSequenceNumber=6460015575321227402, timestamp=1552991928992, source='RCaDMWcA'}, yxZjoXOD=RetentionLease{id='yxZjoXOD', retainingSequenceNumber=6274212870258453507, timestamp=1552991928992, source='QeMpImcA'}, VyiIjJYs=RetentionLease{id='VyiIjJYs', retainingSequenceNumber=2133028230013912907, timestamp=1552991928992, source='yQsjFtqy'}, sfjfrVor=RetentionLease{id='sfjfrVor', retainingSequenceNumber=8701391374091472416, timestamp=1552991928992, source='uFWFHpJw'}, pYQTNwQT=RetentionLease{id='pYQTNwQT', retainingSequenceNumber=4919630631661594402, timestamp=1552991928992, source='hdrfFeBC'}, oWrKJWSr=RetentionLease{id='oWrKJWSr', retainingSequenceNumber=7602602061015029303, timestamp=1552991928992, source='ZvAAwqpt'}}>