[CI] IndicesStoreIntegrationTests.indexCleanup

[brwe]

http://build-us-00.elasticsearch.org/job/es_core_1x_regression/1415/

Related to #9985

[kimchy]

I looked at it, not obvious..., seems like the shard got moved, but ended up not being deleted. Seems unrelated so far to my change.

[brwe]

Here is what happens:
When we delete shards we first check if all expected copies of this shard are actually up and running somewhere else (IndicesStore.deleteShardIfExistElseWhere()). This check checks for STARTED and RELOCATED. In the test we relocate a shard from one node (node_1) to another (node_3) but at the moment node_1 checks if enough shards available the shard is still in POST_RECOVERY on node_3 and therefore the check fails.

This is easily reproducible by adding an artificial delay in IndexShardState.changeState.

Chatted with @s1monw about it and will make a PR to schedule a retry of the check in case the check runs into shards which are in POST_RECOVERY.