Default delayed allocation timeout to 1m from 0

Change the default delayed allocation timeout from 0 (no delayed allocation) to 1m. The value came from a test of having a node with 50 shards being indexed into (so beefy translog requiring flush on shutdown), then shutting it down and starting it back up and waiting for it to join the cluster. This took, on a slow machine, about 30s.
The value is conservatively low and does not try to address a virtual machine / OS restart for now, in order to not have the affect of node going away and users being concerned that shards are not being allocated to the rest of the cluster as a result of that. The setting can always be changed in order to increase the delayed allocation if needed.
closes opensearch-project#12166

core/src/main/java/org/elasticsearch/cluster/routing/UnassignedInfo.java

@@ -42,7 +42,7 @@ public class UnassignedInfo implements ToXContent, Writeable<UnassignedInfo> {
     public static final FormatDateTimeFormatter DATE_TIME_FORMATTER = Joda.forPattern("dateOptionalTime");
 
     public static final String INDEX_DELAYED_NODE_LEFT_TIMEOUT_SETTING = "index.unassigned.node_left.delayed_timeout";
-    private static final TimeValue DEFAULT_DELAYED_NODE_LEFT_TIMEOUT = TimeValue.timeValueMillis(0);
+    private static final TimeValue DEFAULT_DELAYED_NODE_LEFT_TIMEOUT = TimeValue.timeValueMinutes(1);
 
     /**
      * Reason why the shard is in unassigned state.

core/src/test/java/org/elasticsearch/test/InternalTestCluster.java

@@ -50,6 +50,7 @@
 import org.elasticsearch.cluster.node.DiscoveryNodes;
 import org.elasticsearch.cluster.routing.OperationRouting;
 import org.elasticsearch.cluster.routing.ShardRouting;
+import org.elasticsearch.cluster.routing.UnassignedInfo;
 import org.elasticsearch.cluster.routing.allocation.decider.DiskThresholdDecider;
 import org.elasticsearch.cluster.routing.allocation.decider.ThrottlingAllocationDecider;
 import org.elasticsearch.common.Nullable;
@@ -482,6 +483,9 @@ private static Settings getRandomNodeSettings(long seed) {
             builder.put(ScriptService.SCRIPT_CACHE_EXPIRE_SETTING, TimeValue.timeValueMillis(RandomInts.randomIntBetween(random, 750, 10000000)));
         }
 
+        // always default delayed allocation to 0 to make sure we have tests are not delayed
+        builder.put(UnassignedInfo.INDEX_DELAYED_NODE_LEFT_TIMEOUT_SETTING, 0);
+
         return builder.build();
     }
 

docs/reference/index-modules/allocation/delayed.asciidoc

@@ -32,7 +32,7 @@ requests) which have been automatically <<indices-synced-flush,sync-flushed>>.
 
 The allocation of replica shards which become unassigned because a node has
 left can be delayed with the `index.unassigned.node_left.delayed_timeout`
-dynamic setting, which defaults to `0` (reassign shards immediately).
+dynamic setting, which defaults to `1m`.
 
 This setting can be updated on a live index (or on all indices):