[CI] SLMSnapshotBlockingIntegTests.testSnapshotInProgress failure on 7.8

[cbuescher]

Build scan:

https://gradle-enterprise.elastic.co/s/a7updrlkekkgk

Repro line:

./gradlew ':x-pack:plugin:ilm:test' --tests "org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testSnapshotInProgress" \
  -Dtests.seed=B3ED5F3852E51B6C \
  -Dtests.security.manager=true \
  -Dtests.locale=be-BY \
  -Dtests.timezone=Africa/Ouagadougou \
  -Druntime.java=8

Reproduces locally?:

Not after a couple of runs.

Applicable branches:

7.8

Failure history:

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-90d,mode:quick,to:now))&_a=(columns:!(test.failed-testcases),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:b646ed00-7efc-11e8-bf69-63c8ef516157,key:test.failed-testcases.test,negate:!f,params:(query:testSLMRetentionAfterRestore,type:phrase),type:phrase,value:testSLMRetentionAfterRestore),query:(match:(test.failed-testcases.test:(query:testSLMRetentionAfterRestore,type:phrase))))),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:''),sort:!(process.time-start,desc))

Failure excerpt:

java.lang.AssertionError: cluster state size does not match expected:<3111> but was:<3083>
	at __randomizedtesting.SeedInfo.seed([B3ED5F3852E51B6C:A525196DCE9EA62]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.elasticsearch.test.ESIntegTestCase.ensureClusterStateConsistency(ESIntegTestCase.java:1079)
	at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:580)
	at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2218)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[cbuescher]

Another instance on 7.x: https://gradle-enterprise.elastic.co/s/jqc4u3encwa76

[original-brownbear]

The initial issue reported here seems to be the result of a SnapshotLifecycleStats instance that is part of the current cluster state being mutated and thus failing the CS consistency checks.
The fix here is to make SnapshotLifecycleStats immutable as it is part of the cluster state. I don't really have time to handle this right away (don't know the SLM codebase well enough to quickly pull this off). I'll unassign myself for now, maybe someone else can pick it up .

[original-brownbear]

Fixing for the second failure mode in #59140 (comment) is in #59218 , addressing the mutability of the CS is still open here though