Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Supervisord to manage processes #402

Closed
wants to merge 49 commits into from

Conversation

Luatan
Copy link
Contributor

@Luatan Luatan commented Feb 17, 2024

Context

Choices

  • Using Supervisord to manage the processes.
  • Adds Triggers for Discord Notifications tied to the application state of the palworld server
  • Adds a separate script for starting the server, to only run start.sh once on container start.
  • Update and reboot scripts use supervisord to shutdown and restart the server
  • Restore Script uses supervisord to shutdown and restart the server

Processes Managed by Supervisord

  • Palworld
  • Palworld-Event-Handler
  • Supercronic

Test instructions

Environment Variables

These Environment variables can be used for all test cases

DISCORD_WEBHOOK_URL: "your test webhook"
UPDATE_ON_BOOT: false
BACKUP_ENABLED: false
PLAYER_LOGGING_POLL_PERIOD: 10

Always Running Services

These services should always be running if enabled in the config

  • Palworld-Server
  • Crontab
  • Player Logging (can be disabled)

Test

  1. Start Container
  2. Verify that the Server logs correctly to docker logs
  3. Verify that the discord notifications are being sent correctly
  4. Check if all mentioned above are processes are running docker exec -it palworld-server supervisorctl status
  5. Join the server with an palworld client
  6. Verify that the player_logging scripts sends a message as broadcast and in the logs
  7. Stop the container
  8. Verify with docker logs in another terminal that the server stops correctly
  9. Verify that the discord notifications are being sent correctly

Scripts

These scripts use supervisord to shutdown the server if necessary.

  • Restore
  • Update
  • Auto Reboot

Restore Test

  1. Start the container
  2. Create a backup
  3. restore the backup with the command docker exec -it palworld-server restore
  4. Check the logs and verify that the server shutdown and the restore is successfull

Update Test

  1. Start the container
  2. Run the command docker exec palworld-server update and verify that the server is up to date
  3. Change the "manifest" value in palworld\steamapps\appmanifest_2394010.acf on Line 31 to a different value
  4. Run the command docker exec palworld-server update and verify that an update is found
  5. Verify that the server is being shutdown without stopping the container
  6. Verify that the server is started again after steam has completed the "update"

Auto Reboot Test

  1. Start the container
  2. Run the command docker exec palworld-server auto_reboot.sh
  3. Verify that the server shutsdown without stopping the container
  4. Verify that the server starts again

Checklist before requesting a review

  • I have performed a self-review of my code
  • I've added documentation about this change to the README.
  • I've not introduced breaking changes.

Luatan and others added 30 commits February 3, 2024 02:30
@thijsvanloef thijsvanloef added the enhancement New feature or request label Feb 28, 2024
@xHyperElectric
Copy link
Contributor

What's left to do on this?
@Luatan

@Luatan
Copy link
Contributor Author

Luatan commented Mar 5, 2024

What's left to do on this? @Luatan

I didn't really had the time to continue with this. But I am currently struggling with the eventlistener. Since I have added another process (the player logging) the behavior is really strange. If the player logging is enabled it does not send the same events

to see it you can add the line

LogWarn "$header" >&2

to the palworld_state_handler.sh before the if statement which checks the service name

then if I enable the player logging in the docker compose file I get this output:

palworld-server  | Compiling PalWorldSettings.ini done!
palworld-server  | ****GENERATING CRONTAB****
palworld-server  | Player Logging enabled
palworld-server  | time="2024-03-05T19:52:27Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_STARTING len:78
palworld-server  | *********STARTING SERVER*********
palworld-server  | processname:player_logging groupname:player_logging from_state:STOPPED tries:0ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:36ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:78
palworld-server  | processname:player_logging groupname:player_logging from_state:STARTING pid:39ver:3.0 server:supervisor serial:6 pool:palworld-events poolserial:6 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | Shutdown handler: initalize.
palworld-server  | - Existing per-process limit (soft=1048576, hard=1048576) is enough for us (need only 1048576)
palworld-server  | Increasing per-process limit of core file size to infinity.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | Waiting for RCON port to open to show player logging...
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:37ver:3.0 server:supervisor serial:7 pool:palworld-events poolserial:7 eventname:PROCESS_STATE_RUNNING len:66
palworld-server  | Waiting for RCON port to open to show player logging...

and if it is disabled:

palworld-server  | ****GENERATING CRONTAB****
palworld-server  | time="2024-03-05T19:55:16Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | *********STARTED SERVER*********
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:35ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | Shutdown handler: initalize.
palworld-server  | - Existing per-process limit (soft=1048576, hard=1048576) is enough for us (need only 1048576)
palworld-server  | Increasing per-process limit of core file size to infinity.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:36ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:66

But we should get both the events:

  1. Starting
  2. Started

on shutdown there is a similar issue:

  1. Stopping
  2. Stopped

I don't know what I am doing wrong. I think the easiest way is to test it with webbooks. You can see it there very clearly.

If that issue is resolved. The

  • Restore
  • Update
  • Auto Reboot
    scripts need to be changed to use supervisord to shutdown and start the server

@thijsvanloef thijsvanloef added the help wanted Extra attention is needed label Mar 5, 2024
@MusclePr
Copy link
Contributor

MusclePr commented May 20, 2024

@Luatan This solved it.

diff --git a/services/listener/palworld_state_handler.sh b/services/listener/palworld_state_handler.sh
index b8b103a..c61d5bf 100644
--- a/services/listener/palworld_state_handler.sh
+++ b/services/listener/palworld_state_handler.sh
@@ -21,6 +21,12 @@ while true; do
     printf "READY\n"
     read -r header
     parse_headers "$header"
+    LogWarn "${header}" >&2
+    if [ -n "${len}" ]; then
+        read -n "${len}" data
+        LogWarn "${data}" >&2
+        parse_headers "${data}"
+    fi
 
     if [ "${processname}" = "palworld" ]; then
         case $eventname in

ENABLE_PLAYER_LOGGING=true

palworld-server  | ****EXECUTING USERMOD****
palworld-server  | usermod: no changes
palworld-server  | ****Checking for new update****
palworld-server  | Current Version: 3872500952532478729
palworld-server  | The Server is up to date!
palworld-server  | ****GENERATING CONFIG****
palworld-server  | Using Env vars to create PalWorldSettings.ini
palworld-server  | ****Compiling PalWorldSettings.ini****
palworld-server  | Compiling PalWorldSettings.ini done!
palworld-server  | ****GENERATING CRONTAB****
palworld-server  | BACKUP_ENABLED=true
palworld-server  | Adding cronjob for auto backups
palworld-server  | Player Logging enabled
palworld-server  | time="2024-05-20T02:32:37Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | The file already exists: /palworld/Pal/Binaries/Linux/steamclient.so
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0
palworld-server  | *********STARTING SERVER*********
palworld-server  | ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_STARTING len:78
palworld-server  | processname:player_logging groupname:player_logging from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:51
palworld-server  | ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:78
palworld-server  | processname:player_logging groupname:player_logging from_state:STARTING pid:54
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | ver:3.0 server:supervisor serial:6 pool:palworld-events poolserial:6 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:52
palworld-server  | Waiting for RCON port to open to show player logging...
palworld-server  | ver:3.0 server:supervisor serial:7 pool:palworld-events poolserial:7 eventname:PROCESS_STATE_RUNNING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STARTING pid:53
palworld-server  | *********STARTED SERVER*********

ENABLE_PLAYER_LOGGING=false

palworld-server  | ****EXECUTING USERMOD****
palworld-server  | usermod: no changes
palworld-server  | ****Checking for new update****
palworld-server  | Current Version: 3872500952532478729
palworld-server  | The Server is up to date!
palworld-server  | ****GENERATING CONFIG****
palworld-server  | Using Env vars to create PalWorldSettings.ini
palworld-server  | ****Compiling PalWorldSettings.ini****
palworld-server  | Compiling PalWorldSettings.ini done!
palworld-server  | ****GENERATING CRONTAB****
palworld-server  | BACKUP_ENABLED=true
palworld-server  | Adding cronjob for auto backups
palworld-server  | time="2024-05-20T02:34:56Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | The file already exists: /palworld/Pal/Binaries/Linux/steamclient.so
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0
palworld-server  | *********STARTING SERVER*********
palworld-server  | ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:50
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:51
palworld-server  | ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STARTING pid:52
palworld-server  | *********STARTED SERVER*********

@Luatan
Copy link
Contributor Author

Luatan commented May 27, 2024

Hey @MusclePr, thanks for helping. Still need to test it, but I was wondering if this feature is something that should still be implemented. I haven't had the time to really follow the progress of the container. And would like to get some feedback first, before I go ahead and invest more time into this.
Would you mind giving your take about this @thijsvanloef .

@thijsvanloef
Copy link
Owner

Hi @Luatan, I am still not really sure, I'll think about it and get back to you, sorry I was not able to check this out sooner.

@thijsvanloef
Copy link
Owner

Deciding to close this issue. Supervisored is not the way I want to move forward with managing processes. Instead opening a feature request to handle the processes with s6 overlay

@Luatan
Copy link
Contributor Author

Luatan commented Jun 14, 2024

Deciding to close this issue. Supervisored is not the way I want to move forward with managing processes. Instead opening a feature request to handle the processes with s6 overlay

I was actually also looking for alternatives, but couldn't find any. No clue how I didn't find s6 overlay. it looks promising as far as I have seen. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove the need of restart policy for auto update and auto reboot
4 participants