[BUG] org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT > testPendingTasksWithClusterNotRecoveredBlock FAILED

[dblock]

org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT > testPendingTasksWithClusterNotRecoveredBlock FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([F19D942E1E4110F4:5F2076E19A98425]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:979)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:918)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:907)
        at org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT.testPendingTasksWithClusterNotRecoveredBlock(PendingTasksBlocksIT.java:111)

#3259 (comment)

[dbwiddis]

This is one of a few different tests where I've seen an unreadable (truncated with large length) deprecation log message in the failure trace:

[2022-12-29T09:26:34,355][INFO ][o.o.a.a.c.t.PendingTasksBlocksIT] [testPendingTasksWithClusterNotRecoveredBlock] [PendingTasksBlocksIT#testPendingTasksWithClusterNotRecoveredBlock]: setting up test
[2022-12-29T09:26:34,536][DEPRECATION][o.o.d.c.s.Settings       ] [node_s0] [index.force_memory_term_dictionary] setting was dep
...[truncated 1376351 chars]...
eS8W-VsYrMCZy5A}{Ur513a1IRy24HkBPUMqLPw}{127.0.0.1}{127.0.0.1:42445}{imr}{shard_indexing_pressure_enabled=true}}, term: 4, version: 14, reason: ApplyCommitRequest{term=4, version=14, sourceNode={node_s0}

The setting comes from Store.java:

OpenSearch/server/src/main/java/org/opensearch/index/store/Store.java

Lines 151 to 162 in 57d4485

	/**
	* This is an escape hatch for lucenes internal optimization that checks if the IndexInput is an instance of ByteBufferIndexInput
	* and if that's the case doesn't load the term dictionary into ram but loads it off disk iff the fields is not an ID like field.
	* Since this optimization has been added very late in the release processes we add this setting to allow users to opt-out of
	* this by exploiting lucene internals and wrapping the IndexInput in a simple delegate.
	*/
	public static final Setting<Boolean> FORCE_RAM_TERM_DICT = Setting.boolSetting(
	"index.force_memory_term_dictionary",
	false,
	Property.IndexScope,
	Property.Deprecated
	);

Flakiness is introduced as part of setRandomIndexSettings() in OpenSearchIntegTestCase:

OpenSearch/test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java

Lines 474 to 476 in 57d4485

	if (random.nextBoolean()) {
	builder.put(Store.FORCE_RAM_TERM_DICT.getKey(), true);
	}

But similar messages are on completely different log messages on completely unrelated tests:

[2022-11-23T10:30:18,307][INFO ][o.
...[truncated 1171668 chars]...
ansport.TransportService.sendRequest(TransportService.java:718) [main/:?]
	at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:348) [main/:?]
	at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.start(FollowersChecker.java:336) [main/:?]
	at org.opensearch.cluster.coordination.FollowersChecker.lambda$setCurrentNodes$2(FollowersChecker.java:178) [main/:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) [?:?]

<and more...>