Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] The FE&BE&CN startup script automatically detects that the startup is successful #48793

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+94 −12
Open

[Enhancement] The FE&BE&CN startup script automatically detects that the startup is successful #48793

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f98fa17
Use the /api/health interface to determine that the process has been …
zhuxt2015 Jul 23, 2024
9701908
fix bug
zhuxt2015 Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 47 additions & 5 deletions bin/start_backend.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,9 +171,18 @@ fi

if [ ${RUN_BE} -eq 1 ]; then
pidfile=$PID_DIR/be.pid
process_name="backend"
conf_file="$STARROCKS_HOME/conf/be.conf"
fi
if [ ${RUN_CN} -eq 1 ]; then
pidfile=$PID_DIR/cn.pid
process_name="compute node"
conf_file="STARROCKS_HOME/conf/cn.conf"
fi

http_port=$(grep -v ^# $conf_file |grep be_http_port| sed 's/.*= *//')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this already loaded by export_env_from_conf $STARROCKS_HOME/conf/cn.conf in line 72

if [ -z "$http_port" ]; then
http_port="8040"
fi

if [ -f $pidfile ]; then
Expand Down Expand Up @@ -206,14 +215,47 @@ fi
if [ ${RUN_LOG_CONSOLE} -eq 1 ] ; then
# force glog output to console (stderr)
export GLOG_logtostderr=1
else
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't remove this, it'll break the assumption that crash core will be redirected to be.out/cn.out

# redirect stdout/stderr to ${LOG_FILE}
exec &>> ${LOG_FILE}
fi

echo "start time: $(date), server uptime: $(uptime)"
check_health() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really tricky. I am inclined not to do so in start script. the --daemon is a simplest way to provide a daemonized launched solution, but usually it is not enough in production without watchdog. so A supervisor or systemd-service will be prefered to launch the service without --daemon and additional healthy/probing methods can be applied out of the start script and control whether need to restart/kill the process with supervisorctl restart <fe/be/cn> or systemctl restart <fe/be/cn>.

response=$(curl -s -w "\n%{http_code}" http://127.0.0.1:${http_port}/api/health)
body=$(echo "$response" | sed -e '$d')
status_code=$(echo "$response" | tail -n1)

# check http status code
if [ "$status_code" -eq 200 ]
then
# Parse the JSON response and check the status field
status=$(echo "$body" | sed -n 's/.*"status"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')
if [ "$status" = "OK" ]
then
return 0
else
return 1
fi
else
return 1
fi
}

echo "start time: $(date), server uptime: $(uptime)" >> ${LOG_FILE}
if [ ${RUN_DAEMON} -eq 1 ]; then
nohup ${START_BE_CMD} "$@" </dev/null &
echo "The $process_name process is starting!"
nohup ${START_BE_CMD} "$@" &>/dev/null &
if command -v curl &>/dev/null; then
timeout=30
end_time=$((SECONDS + timeout))
while [ $SECONDS -lt $end_time ]; do
if check_health; then
echo "The $process_name process has started successfully!"
exit 0
fi
sleep 1
done
echo "The startup time is more than $timeout seconds, the $process_name process may not start successfully!"
else
echo "This node does not have the curl command, please manually check whether the $process_name process is successfully started"
fi
else
exec ${START_BE_CMD} "$@" </dev/null
fi
zhuxt2015 marked this conversation as resolved.
Show resolved Hide resolved
54 changes: 47 additions & 7 deletions bin/start_fe.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,18 +217,58 @@ if [ ${RUN_LOG_CONSOLE} -eq 1 ] ; then
fi
# force sys_log_to_console = true
echo -e "\nsys_log_to_console = true" >> $STARROCKS_HOME/conf/fe.conf
else
# redirect all subsequent commands' stdout/stderr into $LOG_FILE
exec &>> $LOG_FILE
fi

echo "using java version $JAVA_VERSION"
echo $final_java_opt
echo "start time: $(date), server uptime: $(uptime)"
echo "using java version $JAVA_VERSION" >> $LOG_FILE
echo $final_java_opt >> $LOG_FILE
echo "start time: $(date), server uptime: $(uptime)" >> $LOG_FILE

process_name="frontend"
conf_file="$STARROCKS_HOME/conf/fe.conf"
http_port=$(grep -v ^# $conf_file | grep http_port | sed 's/.*= *//')
if [ -z "$http_port" ]; then
http_port="8030"
fi

check_health() {
response=$(curl -s -w "\n%{http_code}" http://127.0.0.1:${http_port}/api/health)
body=$(echo "$response" | sed -e '$d')
status_code=$(echo "$response" | tail -n1)

# check http status code
if [ "$status_code" -eq 200 ]
then
# Parse the JSON response and check the status field
status=$(echo "$body" | sed -n 's/.*"status"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')
if [ "$status" = "OK" ]
then
return 0
else
return 1
fi
else
return 1
fi
}

# StarRocksFE java process will write its process id into $pidfile
if [ ${RUN_DAEMON} -eq 1 ]; then
nohup $LIMIT $JAVA $final_java_opt com.starrocks.StarRocksFE ${HELPER} ${HOST_TYPE} "$@" </dev/null &
echo "The $process_name process is starting!"
nohup $LIMIT $JAVA $final_java_opt com.starrocks.StarRocksFE ${HELPER} ${HOST_TYPE} "$@" &> /dev/null &
if command -v curl &>/dev/null; then
timeout=30
end_time=$((SECONDS + timeout))
while [ $SECONDS -lt $end_time ]; do
if check_health; then
echo "The $process_name process has started successfully!"
exit 0
fi
sleep 1
done
echo "The startup time is more than $timeout seconds, the $process_name process may not start successfully!"
else
echo "This node does not have the curl command, please manually check whether the $process_name process is successfully started"
fi
else
exec $LIMIT $JAVA $final_java_opt com.starrocks.StarRocksFE ${HELPER} ${HOST_TYPE} "$@" </dev/null
fi
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most risky bug in this code is:
Potential race condition causing premature script exit.

You can modify the code like this:

# StarRocksFE java process will write its process id into $pidfile
if [ ${RUN_DAEMON} -eq 1 ]; then
    echo "The $process_name process is starting!"
    nohup $LIMIT $JAVA $final_java_opt com.starrocks.StarRocksFE ${HELPER} ${HOST_TYPE} "$@" &> /dev/null &

    pid=$!
    if command -v curl &>/dev/null; then
        timeout=30
        end_time=$((SECONDS + timeout))
        while [ $SECONDS -lt $end_time ]; do
            if check_health; then
                echo "The $process_name process has started successfully!"
                exit 0
            fi
            sleep 1
        done
        echo "The startup time is more than $timeout seconds, the $process_name process may not start successfully!"
        kill $pid # Ensure the process does not linger if startup fails
    else
        echo "This node does not have the curl command, please manually check whether the $process_name process is successfully started"
    fi
else
    exec $LIMIT $JAVA $final_java_opt com.starrocks.StarRocksFE ${HELPER} ${HOST_TYPE} "$@" </dev/null
fi

Loading