Random crash of FPM master process in fpm_stdio_child_said

[triplesixman]

Description

Hello,

Since the switch to php8.0-fpm, I have random php crashes which force me to restart it manually and cause a 502 bad gateaway error on my site.

The crashes are random and sometimes occur after several days.

The logs indicate this:

May 8 16:21:14 server kernel: [817145.478932] php-fpm8.0[3482145]: segfault at 105 ip 000055c8afc40309 sp 00007ffd430d3860 error 4 in php-fpm8.0[55c8af9c1000+285000]
May 8 16:21:14 server kernel: [817145.550172] Code: ac 4c 8b a0 f0 00 00 00 41 0f 94 c5 89 4d a8 0f 84 e6 02 00 00 48 8b 45 a0 48 83 c0 70 48 89 45 88 4d 85 e4 0f 84 2d 03 00 00 <41> f6 44 24 05 04 75 11 48 8b 45 a0 8b 80 e4 00 00 00 41 39 44 24
May 8 16:21:14 server systemd[1]: php8.0-fpm.service: Main process exited, code=killed, status=11/SEGV
May 8 16:21:14 server systemd[1]: php8.0-fpm.service: Failed with result 'signal'.
May 8 16:21:14 server systemd[1]: php8.0-fpm.service: Consumed 3d 57min 21.700s CPU time.

Unfortunately I don't have more information and it is impossible to identify the problem. Do you know how I can get more details?

I have thousands of active users on my site and it will not be possible for me to reproduce the crash by myself.

Thank you

PHP Version

PHP 8.0.18

Operating System

Debian 11

[Girgias]

Possible duplicate of #8494 ?

[cmb69]

Do you know how I can get more details?

I stack backtrace might be helpful.

[iluuu1994]

@triplesixman Would it be possible to generate a backtrace? Otherwise we have pretty much no chance of finding the actual issue.

[Poulpatine]

Hi,

Sorry for the delay we finally been able to reproduce the issue and to gather a backtrace:

gdb) bt
#0  fpm_stdio_child_said (ev=0x55bf8d3901f0, which=2, arg=0x55bf8d390180) at ./sapi/fpm/fpm/fpm_stdio.c:156
#1  0x000055bf8cc1ef66 in fpm_event_epoll_wait (queue=<optimized out>, timeout=<optimized out>) at ./sapi/fpm/fpm/events/epoll.c:143
#2  0x000055bf8cc11a85 in fpm_event_loop (err=err@entry=0) at ./sapi/fpm/fpm/fpm_events.c:427
#3  0x000055bf8cc0bea7 in fpm_run (max_requests=max_requests@entry=0x7ffefea58c3c) at ./sapi/fpm/fpm/fpm.c:113
#4  0x000055bf8c9d40f8 in main (argc=4, argv=0x7ffefea59178) at ./sapi/fpm/fpm/fpm_main.c:1828

[iluuu1994]

@bukka Looks like log_stream here is NULL (or an invalid pointer).

php-src/sapi/fpm/fpm/fpm_stdio.c

Line 156 in 10dc522

if (log_stream->is_stdout != (unsigned int)is_stdout || log_stream->child_pid != (int)child->pid) {

I'm completely unfamiliar with fpm, do you know how this could happen?

[bukka]

I don't see anything suspicious from looking to the code. If it was NULL, then it would fall to the different branch and was created so it would need to be some invalid pointer but again that should not happen based on the code. The only thing that comes to my mind is some sort of memory corruption somewhere. I will add this on my list and will try to do more careful review.

@Poulpatine could you share your FPM config?

[Poulpatine]

Sure, here it is.
It's the Debian default configuration, just modified to be able to generate coredumps.

;;;;;;;;;;;;;;;;;;;;;
; FPM Configuration ;
;;;;;;;;;;;;;;;;;;;;;

; All relative paths in this configuration file are relative to PHP's install
; prefix (/usr). This prefix can be dynamically changed by using the
; '-p' argument from the command line.

;;;;;;;;;;;;;;;;;;
; Global Options ;
;;;;;;;;;;;;;;;;;;

[global]
; Pid file
; Note: the default prefix is /var
; Default Value: none
; Warning: if you change the value here, you need to modify systemd
; service PIDFile= setting to match the value here.
pid = /run/php/php8.0-fpm.pid

; Error log file
; If it's set to "syslog", log is sent to syslogd instead of being written
; into a local file.
; Note: the default prefix is /var
; Default Value: log/php-fpm.log
error_log = /var/log/php8.0-fpm.log

; syslog_facility is used to specify what type of program is logging the
; message. This lets syslogd specify that messages from different facilities
; will be handled differently.
; See syslog(3) for possible values (ex daemon equiv LOG_DAEMON)
; Default Value: daemon
;syslog.facility = daemon

; syslog_ident is prepended to every message. If you have multiple FPM
; instances running on the same server, you can change the default value
; which must suit common needs.
; Default Value: php-fpm
;syslog.ident = php-fpm

; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
;log_level = notice

; Log limit on number of characters in the single line (log entry). If the
; line is over the limit, it is wrapped on multiple lines. The limit is for
; all logged characters including message prefix and suffix if present. However
; the new line character does not count into it as it is present only when
; logging to a file descriptor. It means the new line character is not present
; when logging to syslog.
; Default Value: 1024
;log_limit = 4096

; Log buffering specifies if the log line is buffered which means that the
; line is written in a single write operation. If the value is false, then the
; data is written directly into the file descriptor. It is an experimental
; option that can potentionaly improve logging performance and memory usage
; for some heavy logging scenarios. This option is ignored if logging to syslog
; as it has to be always buffered.
; Default value: yes
;log_buffering = no

; If this number of child processes exit with SIGSEGV or SIGBUS within the time
; interval set by emergency_restart_interval then FPM will restart. A value
; of '0' means 'Off'.
; Default Value: 0
;emergency_restart_threshold = 0

; Interval of time used by emergency_restart_interval to determine when
; a graceful restart will be initiated.  This can be useful to work around
; accidental corruptions in an accelerator's shared memory.
; Available Units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
;emergency_restart_interval = 0

; Time limit for child processes to wait for a reaction on signals from master.
; Available units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
;process_control_timeout = 0

; The maximum number of processes FPM will fork. This has been designed to control
; the global number of processes when using dynamic PM within a lot of pools.
; Use it with caution.
; Note: A value of 0 indicates no limit
; Default Value: 0
; process.max = 128

; Specify the nice(2) priority to apply to the master process (only if set)
; The value can vary from -19 (highest priority) to 20 (lowest priority)
; Note: - It will only work if the FPM master process is launched as root
;       - The pool process will inherit the master process priority
;         unless specified otherwise
; Default Value: no set
; process.priority = -19

; Send FPM to background. Set to 'no' to keep FPM in foreground for debugging.
; Default Value: yes
;daemonize = yes

; Set open file descriptor rlimit for the master process.
; Default Value: system defined value
;rlimit_files = 1024

; Set max core size rlimit for the master process.
; Possible Values: 'unlimited' or an integer greater or equal to 0
; Default Value: system defined value
;rlimit_core = 0
rlimit_core = unlimited

; Specify the event mechanism FPM will use. The following is available:
; - select     (any POSIX os)
; - poll       (any POSIX os)
; - epoll      (linux >= 2.5.44)
; - kqueue     (FreeBSD >= 4.1, OpenBSD >= 2.9, NetBSD >= 2.0)
; - /dev/poll  (Solaris >= 7)
; - port       (Solaris >= 10)
; Default Value: not set (auto detection)
;events.mechanism = epoll

; When FPM is built with systemd integration, specify the interval,
; in seconds, between health report notification to systemd.
; Set to 0 to disable.
; Available Units: s(econds), m(inutes), h(ours)
; Default Unit: seconds
; Default value: 10
;systemd_interval = 10

;;;;;;;;;;;;;;;;;;;;
; Pool Definitions ;
;;;;;;;;;;;;;;;;;;;;

; Multiple pools of child processes may be started with different listening
; ports and different management options.  The name of the pool will be
; used in logs and stats. There is no limitation on the number of pools which
; FPM can handle. Your system will tell you anyway :)

; Include one or more files. If glob(3) exists, it is used to include a bunch of
; files from a glob(3) pattern. This directive can be used everywhere in the
; file.
; Relative path can also be used. They will be prefixed by:
;  - the global prefix if it's been set (-p argument)
;  - /usr otherwise
include=/etc/php/8.0/fpm/pool.d/*.conf

And /etc/php/8.0/fpm/pool.d/www.conf :

[www]
user = www-data
group = www-data
listen = 127.0.0.1:9000
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 400
pm.start_servers = 60
pm.min_spare_servers = 40
pm.max_spare_servers = 70
pm.max_requests = 20000
request_terminate_timeout = 3600
request_slowlog_timeout = 300s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes
pm.status_path = /status
slowlog = /var/log/php/slowlog-site.log

php_admin_value[error_log] = /var/log/php/php7.2-fpm.log
php_admin_flag[log_errors] = off