[apache#1202] improvement: Add HealthScriptChecker for execute specia…

…l health check shell script (apache#1203)

### What changes were proposed in this pull request?

add HealthScriptChecker for execute special health check shell script
ref: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/health/NodeHealthScriptRunner.java

### Why are the changes needed?

Fix:  apache#1202

### Does this PR introduce _any_ user-facing change?
`rss.server.health.checker.script.path` : The health script file path for HealthScriptChecker, if script file should have execute permission.

`rss.server.health.checker.script.execi.timeout`: The health script file execute timeout seconds


### How was this patch tested?
add ut

---------

Co-authored-by: jam.xu <jam.xu@vipshop.com>

docs/server_guide.md

@@ -107,7 +107,8 @@ This document will introduce how to deploy Uniffle shuffle servers.
 |rss.server.storageMediaProvider.from.env.key|-| Sometimes, the local storage type/media info is provided by external system. RSS would read the env key defined by this configuration and get info about the storage media of its basePaths |
 |rss.server.decommission.check.interval|60000| The interval(ms) to check if all applications have finish when server is decommissioning                                                                                                        |
 |rss.server.decommission.shutdown|true| Whether shutdown the server after server is decommissioned                                                                                                                                  |
-
+|rss.server.health.checker.script.path| - | The health script path for `HealthScriptChecker`. To enable `HealthScriptChecker`, need to set `rss.server.health.checker.class.names` and set `rss.server.health.check.enable` to true.|
+|rss.server.health.checker.script.execute.timeout| 5000 | Timeout for `HealthScriptChecker` execute health script.(ms)|
 
 ### Huge Partition Optimization
 A huge partition is a common problem for Spark/MR and so on, caused by data skew. And it can cause the shuffle server to become unstable. To solve this, we introduce some mechanisms to limit the writing of huge partitions to avoid affecting regular partitions, more details can be found in [ISSUE-378](https://github.com/apache/incubator-uniffle/issues/378). The basic rules for limiting large partitions are memory usage limits and flushing individual buffers directly to persistent storage.

server/src/main/java/org/apache/uniffle/server/HealthScriptChecker.java

@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.server;
+
+import org.apache.hadoop.util.NodeHealthScriptRunner;
+import org.apache.hadoop.util.Shell;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.uniffle.common.exception.RssException;
+
+public class HealthScriptChecker extends Checker {
+  private static final Logger LOG = LoggerFactory.getLogger(HealthScriptChecker.class);
+  private String healthScriptPath;
+  private static final String ERROR_PATTERN = "ERROR";
+  private long scriptTimeout;
+
+  HealthScriptChecker(ShuffleServerConf conf) {
+    super(conf);
+    this.healthScriptPath = conf.getString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH);
+    if (!NodeHealthScriptRunner.shouldRun(healthScriptPath)) {
+      LOG.error(
+          "Rss health check script:"
+              + healthScriptPath
+              + " is not available "
+              + "or doesn't have execute permission, so abort server.");
+      throw new RssException("Health script not available.");
+    }
+    this.scriptTimeout = conf.getLong(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_EXECUTE_TIMEOUT);
+  }
+
+  @Override
+  public boolean checkIsHealthy() {
+    HealthCheckerExitStatus status = HealthCheckerExitStatus.SUCCESS;
+    // always build executor is to load the latest script, the script file can update in need.
+    Shell.ShellCommandExecutor commandExecutor =
+        new Shell.ShellCommandExecutor(new String[] {healthScriptPath}, null, null, scriptTimeout);
+    try {
+      commandExecutor.execute();
+    } catch (Shell.ExitCodeException e) {
+      // ignore the exit code of the script
+      status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
+      // On Windows, we will not hit the Stream closed IOException
+      // thrown by stdout buffered reader for timeout event.
+      if (Shell.WINDOWS && commandExecutor.isTimedOut()) {
+        status = HealthCheckerExitStatus.TIMED_OUT;
+      }
+    } catch (Exception e) {
+      LOG.warn("execute health script exception, please check script.", e);
+      if (!commandExecutor.isTimedOut()) {
+        status = HealthCheckerExitStatus.FAILED_WITH_EXCEPTION;
+      } else {
+        status = HealthCheckerExitStatus.TIMED_OUT;
+      }
+    } finally {
+      if (status == HealthCheckerExitStatus.SUCCESS) {
+        if (hasErrors(commandExecutor.getOutput())) {
+          status = HealthCheckerExitStatus.FAILED;
+        }
+      }
+    }
+    if (status != HealthCheckerExitStatus.SUCCESS) {
+      LOG.warn("health script check failed. exit status : " + status);
+    }
+    return status == HealthCheckerExitStatus.SUCCESS;
+  }
+
+  private boolean hasErrors(String output) {
+    String[] splits = output.split("\n");
+    for (String split : splits) {
+      if (split.startsWith(ERROR_PATTERN)) {
+        return true;
+      }
+    }
+    return false;
+  }
+
+  private enum HealthCheckerExitStatus {
+    SUCCESS,
+    TIMED_OUT,
+    FAILED_WITH_EXIT_CODE,
+    FAILED_WITH_EXCEPTION,
+    FAILED
+  }
+}

server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java

@@ -271,6 +271,19 @@ public class ShuffleServerConf extends RssBaseConf {
           .noDefaultValue()
           .withDescription("The list of the Checker's name");
 
+  public static final ConfigOption<String> HEALTH_CHECKER_SCRIPT_PATH =
+      ConfigOptions.key("rss.server.health.checker.script.path")
+          .stringType()
+          .noDefaultValue()
+          .withDescription(
+              "The health script file path for HealthScriptChecker, if script file should have execute permission.");
+
+  public static final ConfigOption<Long> HEALTH_CHECKER_SCRIPT_EXECUTE_TIMEOUT =
+      ConfigOptions.key("rss.server.health.checker.script.execute.timeout")
+          .longType()
+          .defaultValue(5000L)
+          .withDescription("The health script file execute timeout ms.");
+
   public static final ConfigOption<Double> SERVER_MEMORY_SHUFFLE_LOWWATERMARK_PERCENTAGE =
       ConfigOptions.key("rss.server.memory.shuffle.lowWaterMark.percentage")
           .doubleType()

server/src/test/java/org/apache/uniffle/server/HealthScriptCheckerTest.java

@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.server;
+
+import java.io.File;
+import java.util.Objects;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class HealthScriptCheckerTest {
+  private String healthScriptTemplate1;
+  private String healthScriptTemplate2;
+  private String healthScriptTemplate3;
+  private String healthScriptTemplate4;
+
+  @BeforeEach
+  void setUp() {
+    healthScriptTemplate1 = getScriptFilePath("healthy-script1.sh");
+    healthScriptTemplate2 = getScriptFilePath("healthy-script2.sh");
+    healthScriptTemplate3 = getScriptFilePath("healthy-script3.sh");
+    healthScriptTemplate4 = getScriptFilePath("healthy-script4.sh");
+    setExecutable(healthScriptTemplate1);
+    setExecutable(healthScriptTemplate2);
+    setExecutable(healthScriptTemplate3);
+    setExecutable(healthScriptTemplate4);
+  }
+
+  @Test
+  void checkIsHealthy() {
+    ShuffleServerConf conf = new ShuffleServerConf();
+    conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH, healthScriptTemplate1);
+    HealthScriptChecker checker = new HealthScriptChecker(conf);
+    assertTrue(checker.checkIsHealthy());
+
+    conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH, healthScriptTemplate2);
+    checker = new HealthScriptChecker(conf);
+    assertFalse(checker.checkIsHealthy());
+
+    conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH, healthScriptTemplate3);
+    checker = new HealthScriptChecker(conf);
+    assertFalse(checker.checkIsHealthy());
+
+    conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH, healthScriptTemplate4);
+    conf.setLong(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_EXECUTE_TIMEOUT, 3000L);
+    checker = new HealthScriptChecker(conf);
+    assertFalse(checker.checkIsHealthy());
+  }
+
+  private String getScriptFilePath(String fileName) {
+    return Objects.requireNonNull(this.getClass().getClassLoader().getResource(fileName)).getFile();
+  }
+
+  private boolean setExecutable(String filePath) {
+    File file = new File(filePath);
+    file.setExecutable(true);
+    return file.canExecute();
+  }
+}

server/src/test/resources/healthy-script1.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+echo "NORMAL: Check Disk is Normal"

server/src/test/resources/healthy-script2.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+echo "ERROR(disk_io): disk with Input/output error"

server/src/test/resources/healthy-script3.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+exit 1

server/src/test/resources/healthy-script4.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+sleep 5s