Motion events can cause zombie processes in some circumstances

[AntiSol]

Did you read the guide?

Yes

What is the base version number of Motion being used?

4.6.x

What was the install method?

Installed via package tool

What is base architecture?

ARM-64bit

What is the distro being used?

Raspbian

Disto version number

bookworm

Camera type(s) being used?

PI cam via libcamerify

Describe the issue/problem and steps to reproduce

As discussed on the mailing list and in this closed issue, processes run by motion when events occur can become zombies in some circumstances.

Just thought I'd raise an issue to make it easier to keep track of, and also to make this more searchable for others having the same issue.

@Mr-Dave Any info on when we might see the fix you mentioned on the mailing list merged in, or a branch with a fix? I'm super keen to see this resolved because it's causing my home security system to be unreliable and miss movement events. I don't see a develop branch or any recent commits. Thanks!

Motion log output at log_level 8

(already confirmed on mailing list, plese LMK if a log is actually needed)

[Mr-Dave]

Just a FYI that upon further review of the Motion code, there is some handling of SIGCHLD and once that was implemented in my test program there were no longer any zombies.

As such, I'm still not able to recreate this issue.

[AntiSol]

are you saying that the script being run on events needs to handle sigchild?
Could you post your test program?

[Mr-Dave]

No. The script does not need to send a signal. The child process automatically sends a SIGCHLD to the parent when it ends. The parent then needs to acknowledge that signal that the child terminated. The test program is in this gist and if the line with the waitpid is removed, zombies will occur.

[AntiSol]

@Mr-Dave Thanks!

I did a bunch of playing around with your test script and I'm also not able to replicate the issue using this script, even when wrapping the test program with libcamerify, and regardless of whether I run it on the raspi or elsewhere.

I had a look through the test program source, the motion source, and libcamera's source, and I'm now thinking that maybe the problem lies in libcamerify somewhere, but it's very difficult for me to track down because I'm really not a C guy.

Could I ask you to take a look around this signal handling in the libcamera code and seeing if you can spot anything? I notice that they're also setting up a SIGCHLD handler here, but if I read this correctly they're taking into account the possibility that a handler might already be set up, and also running that (line ~88).

For context, their libcamerify script uses LD_PRELOAD to load their v4l-compat.so when running motion.

Note that as mentioned, I still don't get zombies when using libcamerify to wrap your test program. I'm thinking maybe there's some condition where libcamera fires an external program and is not handling the child processes' signals correctly? i.e maybe you have to actually use the camera in the test program? like I said, not a C(++) guy, so it's hard for me to follow.

[Mr-Dave]

Try to comment out this line with a //

[AntiSol]

It looks like you're onto something @Mr-Dave!

I commented out that line and compiled, and since then I've had 0 zombie processes in over an hour. Previous to that I was getting about 1 zombie every minute. Very promising!

[AntiSol]

@Mr-Dave OK, now I have some fairly compelling evidence that you've found the cause.

First, I commented out the line you pointed me to, recompiled, and replaced the motion binary with the recompiled version.

Then I monitored my watchdog script which counts zombie processes every 10 mins for a few hours. 0 zombies.

Finally, I removed the commenting from the line, reactivating it, and recompiled again. Zombies immediately appeared.

Here's the output from my watchdog script:

$ ./watchdog.sh
Thu 25 Apr 2024 17:01:16 AEST: 0 zombies
Thu 25 Apr 2024 17:11:16 AEST: 0 zombies
Thu 25 Apr 2024 17:21:16 AEST: 0 zombies
Thu 25 Apr 2024 17:31:16 AEST: 0 zombies
Thu 25 Apr 2024 17:41:16 AEST: 0 zombies
Thu 25 Apr 2024 17:51:16 AEST: 0 zombies
Thu 25 Apr 2024 18:01:16 AEST: 0 zombies
Thu 25 Apr 2024 18:11:17 AEST: 0 zombies
Thu 25 Apr 2024 18:21:17 AEST: 0 zombies
Thu 25 Apr 2024 18:31:17 AEST: 0 zombies
Thu 25 Apr 2024 18:41:17 AEST: 0 zombies
Thu 25 Apr 2024 18:51:17 AEST: 0 zombies
Thu 25 Apr 2024 19:01:17 AEST: 0 zombies
Thu 25 Apr 2024 19:11:17 AEST: 0 zombies
Thu 25 Apr 2024 19:21:17 AEST: 0 zombies
Thu 25 Apr 2024 19:31:17 AEST: 0 zombies
Thu 25 Apr 2024 19:41:17 AEST: 0 zombies
Thu 25 Apr 2024 19:51:17 AEST: 0 zombies
Thu 25 Apr 2024 20:01:17 AEST: 0 zombies
Thu 25 Apr 2024 20:11:17 AEST: 0 zombies
Thu 25 Apr 2024 20:21:17 AEST: 0 zombies
-------------recompiled and restarted with line reactivated---------
Thu 25 Apr 2024 20:31:17 AEST: 2 zombies
Thu 25 Apr 2024 20:41:17 AEST: 14 zombies
Thu 25 Apr 2024 20:51:17 AEST: 24 zombies
Thu 25 Apr 2024 21:01:18 AEST: 34 zombies

[MichaIng]

Many thanks for investigating and a possible solution. I try to understand the background of the issue.

First of all, this waitpid loop. I know it is not directly related to the issue, but good to understand how releasing child process resources works in C:

• waitpid waits for a child process to terminate.
• waitpid(-1 waits for any child process to terminate.
• WNOHANG lets waitpid return immediately, if no child has terminated, and it returns an integer > 0 only, if an actual child has terminated: its PID.
• So while (waitpid(-1, NULL, WNOHANG) > 0) loops until either either no more child has terminated (or an error happens).
• I was wondering what the point of this is, as long as there is nothing done in the loop. But any child, which has been terminated, has to be waited for to release resources, else it will remain in zombie state.
• If I understand it correctly, compared to just waitpid(-1, NULL, 0) (which equals wait()), this is beneficial if there can be multiple child processes. If multiple children were terminated, wait() would just release resources for a single child, leaving all others as zombies, while the loop handles all of them. This explains the behaviour of the test program pretty well.

Now about ignoring SIGCHLD:

• First of all, I do not really understand the code around this. A signal action with SIG_IGN handler is generated and applied to SIGCHLD. Wouldn't signal(SIGCHLD, SIG_IGN); do the same?
• If this really basically ignores SIGCHLD, and SIGCHLD is calling the above waitpid loop, why is this only causing issues in combinations with libcamerify?
• Actually, the signal action is changed to the above loop here: https://github.com/Motion-Project/motion/blob/11e5ca7/src/motion.c#L431-L470
• Checking the main function: https://github.com/Motion-Project/motion/blob/11e5ca7/src/motion.c#L3458-L3468
  • webu_start() is called from motion_startup(), after setup_signals(), so to me it looks like the above SIGCHLD handler loop is assign first, and then it is overwritten with SIG_IGN, which would render the whole sigchild_action setup unused. Am I missing something?
• I tried to find the reason for ignoring SIGCHLD in the first place, but it was already part of the initial code import to GitHub (first commit): https://github.com/Motion-Project/motion/blob/5221670/webhttpd.c
• Probably the SIGCHLD handler is changed again somewhere else back to sigchild_action, and this is somehow prevented by libcamerify?

[AntiSol]

@MichaIng This is a pretty good breakdown and aligns pretty well with my (limited!) understanding, good work writing up this summary :)

this is beneficial if there can be multiple child processes. If multiple children were terminated, wait() would just release resources for a single child, leaving all others as zombies, while the loop handles all of them.

Yes, exactly, this is my understanding too.

Wouldn't signal(SIGCHLD, SIG_IGN); do the same?

I don't think so, because then you're just ignoring the signal, and never wait()ing on the child. from the manual:

if a wait is not performed, then the terminated child remains in a "zombie" state

So ignoring the signal would result in zombies.

IIUC the reason the process hangs around in a zombie state in these circumstances is because the kernel is holding on to it's exit status in case the calling program wants that value.

(Interestingly, the kernel doesn't hold on to very much information about zombies at all, e.g if you examine /proc/<zombie_pid>/cmdline or /proc/<zombie_pid>/exe these will be empty [this made figuring out the command line challenging!])

why is this only causing issues in combinations with libcamerify

My guess, (poorly) outlined above, is that libcamerify is attaching its own handler to the SIGCHLD signal, and that there's some sort of conflict going on here which is resulting in motion's handler never firing and this wait() never happening. But I haven't put in the time to really trace/understand it, and it's very possible that I'm wrong. If someone with a better grasp on all this wanted to explain I would very curious and grateful!

Actually, the signal action is changed to the above loop here

IIUC, setup_signals is the only place where this signal handler for SIGCHLD is set up, other than in the http server. There's no call to signal() in motion itself, that's just a shortcut MrDave used in the test program for the more complicated sigaction construct used in the actual program (in setup_signals).

to me it looks like the above SIGCHLD handler loop is assign first, and then it is overwritten with SIG_IGN, which would render the whole sigchild_action setup unused. Am I missing something?

I think that it's probably forking somewhere before starting the http server, i.e the main motion thread has the signal handlers set up in setup_signals, and then it forks, and then only the http server thread's signal handler is overridden with the SIG_IGN.

Maybe this is why libcamerify causes issues - perhaps it's not accounting for a multithreaded program with different handlers on different threads, and so is firing the http server thread's handler rather than the main thread's?

I tried to find the reason for ignoring SIGCHLD in the first place

Yeah I'm not sure about that either. The SIGPIPE handler (sort of, mostly) makes sense to me because it seems to be handling an issue where e.g the http server tries to read/write after a connection has been broken (search for SIGPIPE here), but I don't understand why you'd trap SIGCHLD there, since it doesn't seem to call any child processes as far as I can see.

If you're examing vcs history to find when it was added, there might be an old svn system somewhere like sourceforge that you could look at?

Probably the SIGCHLD handler is changed again somewhere else back to sigchild_action, and this is somehow prevented by libcamerify?

I think it's that the handler in the main thread is never changed, this is only the http server thread.

---

I'd like to emphasise that I haven't spent a huge amount of time understanding or tracing it all, and a bunch of this is speculation. I'd love to be corrected on any point by anybody who knows better! :)

[MichaIng]

Wouldn't signal(SIGCHLD, SIG_IGN); do the same?

I don't think so, because then you're just ignoring the signal, and never wait()ing on the child. from the manual:

I meant the override in the HTTP server/webu, which intentionally/as documented is to ignore SIGCHLD (and SIGPIPE). Obviously, it does apply for those motion event handler shell processes which your zombies were from, at least when everything is wrapped into libcamerify, since commenting out this override did fix things for you. Probably libcamerify somehow affects the context in which this signal handler changes apply.

@Mr-Dave now just removed those ignore handlers completely. Probably the best when it is not known anymore what the reason for those were, and whether it still applies after 19 years. We should just keep an eye on whether some other unexpected things happen now around motion child processes.

Many thanks to both of you for investigating/testing/fixing (as far as it looks) this issue. Until someone at motionEye finds time to implement MotionPlus support, or for users who want/need to use motion for some other reason, libcamerify is the only way to get the RPi camera module v3 to work, and any camera module since RPi OS Bookworm (where the legacy camera stack has been completely removed).

[Mr-Dave]

The complete removal was based upon looking at the examples from libmicrohttpd. None of the examples from libmicrohttpd nor other code using libmicrohttpd on github revises signal processing.

Based upon this, I should have removed that signal processing when I fully implemented libmicrohttpd and removed the homegrown http code years ago.

(Still don't know why it occurred on some distros but not others)

[AntiSol]

confirmed fixed on 2 pis now. 0 zombies in >24h, no other issues noticed