5.1.6,使用RedisPool高并发出错 ERROR php_swoole_server_rshutdown() (ERRNO 503)

[mdys]

Please answer these questions before submitting your issue.

1. What did you do? If possible, provide a simple script for reproducing the error.
   server.php

$redis_pool = new RedisPool((new RedisConfig)
		        ->withHost('127.0.0.1')
		        ->withPort(6379)
		        ->withAuth('')
		        ->withDbIndex(0)
		        ->withTimeout(1)
		        ,100000
		  );
$this->redis_pool = $redis_pool;
public function onOpen( $serv, $request ) {
    $redis_pool=$this->redis_pool; 
    $redis = $redis_pool->get(); 
    $redis->get('fd_'.$request->fd);
    $redis_pool->put($redis);
}

在websocket onpen事件时，如果有高并发,大约1s 1000个连接，会有一定几率产生错误，1000个连接失败几十个，产生如下错误：

[2024-12-20 03:53:02 *2291984.253]	
ERROR	php_swoole_server_rshutdown() (ERRNO 503): 
Fatal error: Uncaught RedisException: Connection timed out in @swoole/library/core/Database/RedisPool.php:43
Stack trace:
#0 @swoole/library/core/Database/RedisPool.php(43): Redis->connect()
#1 @swoole/library/core/ConnectionPool.php(88): Swoole\Database\RedisPool->Swoole\Database\{closure}()
#2 @swoole/library/core/ConnectionPool.php(54): Swoole\ConnectionPool->make()
#3 /home/wwwroot/html/server.php(876): Swoole\ConnectionPool->get()
#4 /home/wwwroot/html/server.php(1530): serv_s->onOpen()
#5 [internal function]: serv_s->{closure}()
#6 {main}
  thrown in @swoole/library/core/Database/RedisPool.php on line 43
[2024-12-20 03:53:02 #2291153.0]	WARNING	Server::check_worker_exit_status(): worker(pid=2291967, id=236) abnormal exit, status=0, signal=11
A bug occurred in Swoole-v5.1.6, please report it.
The Swoole developers probably don't know about it,
and unless you report it, chances are it won't be fixed.
You can read How to report a bug doc before submitting any bug reports:
>> https://github.com/swoole/swoole-src/blob/master/.github/ISSUE.md 
Please do not send bug reports in the mailing list or personal letters.
The issue page is also suitable to submit feature requests.

OS: Linux 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64
GCC_VERSION: 8.3.1 20190311 (Red Hat 8.3.1-3)
OPENSSL_VERSION: OpenSSL 1.0.2k-fips  26 Jan 2017
PHP_VERSION : 8.3.8

2. What did you expect to see?

3. What did you see instead?

4. What version of Swoole are you using (show your php --ri swoole)?

Swoole => enabled
Author => Swoole Team <team@swoole.com>
Version => 5.1.6
Built => Dec  2 2024 14:14:42
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.0.2k-fips  26 Jan 2017
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
c-ares => 1.10.0
zlib => 1.2.7
brotli => E16777225/D16777225
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled
Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

5. What is your machine environment used (show your uname -a & php -v & gcc -v) ?

PHP 8.3.8 (cli) (built: Jun 4 2024 14:53:17) (NTS gcc x86_64)
Copyright (c) The PHP Group
Zend Engine v4.3.8, Copyright (c) Zend Technologies
with Zend OPcache v8.3.8, Copyright (c), by Zend Technologies`
`COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-8/root/usr --mandir=/opt/rh/devtoolset-8/root/usr/share/man --infodir=/opt/rh/devtoolset-8/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --with-default-libstdcxx-abi=gcc4-compatible --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-8.3.1-20190311/obj-x86_64-redhat-linux/isl-install --disable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)

[mdys]

补充下报错信息：

Dec 20 00:34:47 localhost Swoole_process_server_worker_155: *** Error in `Swoole_process_server_worker_155': corrupted size vs. prev_size: 0x0000562e5f02b780 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_132: *** Error in `Swoole_process_server_worker_132': corrupted size vs. prev_size: 0x0000562e5f0274b0 ***
Dec 20 00:34:47 localhost kernel: php[1769282]: segfault at 7f70148001e0 ip 00007f682ab8da5d sp 00007fff39abb788 error 6 in libc-2.17.so[7f682aafe000+1c4000]
Dec 20 00:34:47 localhost kernel: php[1769501]: segfault at 2148 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769232]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769411]: segfault at 588 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769269]: segfault at 3e8 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769306]: segfault at 138 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769316]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769308]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769335]: segfault at 2148 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost kernel: php[1769278]: segfault at 48 ip 00007f68158c5987 sp 00007fff39abb7f0 error 4 in swoole.so[7f68157cc000+2ce000]
Dec 20 00:34:47 localhost Swoole_process_server_worker_93: *** Error in `Swoole_process_server_worker_93': corrupted size vs. prev_size: 0x0000562e5f032850 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_117: *** Error in `Swoole_process_server_worker_117': corrupted size vs. prev_size: 0x0000562e5f0274b0 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_8: *** Error in `Swoole_process_server_worker_8': corrupted size vs. prev_size: 0x0000562e5f054d80 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_200: *** Error in `Swoole_process_server_worker_200': double free or corruption (!prev): 0x0000562e5f232f20 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_194: *** Error in `Swoole_process_server_worker_194': corrupted size vs. prev_size: 0x0000562e5f02b780 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_345: *** Error in `Swoole_process_server_worker_345': corrupted size vs. prev_size: 0x0000562e5f054620 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_110: *** Error in `Swoole_process_server_worker_110': corrupted size vs. prev_size: 0x0000562e5f0274b0 ***
Dec 20 00:34:47 localhost Swoole_process_server_worker_202: *** Error in `Swoole_process_server_worker_202': double free or corruption (!prev): 0x0000562e5f0456a0 ***
Dec 20 03:28:11 localhost kernel: php             D ffff9b505fb1acc0     0 2218768 2194582 0x00000080
Dec 20 03:28:11 localhost kernel: Call Trace:
Dec 20 03:28:11 localhost kernel: [<ffffffffa4588089>] schedule_preempt_disabled+0x29/0x70
Dec 20 03:28:11 localhost kernel: [<ffffffffa4585ff7>] __mutex_lock_slowpath+0xc7/0x1d0
Dec 20 03:28:11 localhost kernel: [<ffffffffa45853cf>] mutex_lock+0x1f/0x2f
Dec 20 03:28:11 localhost kernel: [<ffffffffa412d206>] ima_file_check+0xa6/0x1b0
Dec 20 03:28:11 localhost kernel: [<ffffffffa405ddda>] do_last+0x59a/0x1340
Dec 20 03:28:11 localhost kernel: [<ffffffffa405ec4d>] path_openat+0xcd/0x5a0
Dec 20 03:28:11 localhost kernel: [<ffffffffa3f11db8>] ? get_futex_key+0x1c8/0x2c0
Dec 20 03:28:11 localhost kernel: [<ffffffffa3eda830>] ? try_to_wake_up+0x190/0x390
Dec 20 03:28:11 localhost kernel: [<ffffffffa4060e9d>] do_filp_open+0x4d/0xb0
Dec 20 03:28:11 localhost kernel: [<ffffffffa406ef97>] ? __alloc_fd+0x47/0x170
Dec 20 03:28:11 localhost kernel: [<ffffffffa404c9e4>] do_sys_open+0x124/0x220
Dec 20 03:28:11 localhost kernel: [<ffffffffa404cafe>] SyS_open+0x1e/0x20
Dec 20 03:28:11 localhost kernel: [<ffffffffa4593f92>] system_call_fastpath+0x25/0x2a
Dec 20 03:28:11 localhost kernel: INFO: task php:2218773 blocked for more than 120 seconds.
Dec 20 03:28:11 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

sysctl.conf

vm.swappiness = 0 
fs.aio-max-nr=1048576
fs.file-max = 9999990
vm.min_free_kbytes = 2097152
vm.max_map_count = 655360
kernel.threads-max=2061206
kernel.pid_max=4194303
kernel.perf_cpu_time_max_percent=3
kernel.perf_event_max_sample_rate=1000
net.ipv4.ip_forward = 1
kernel.msgmnb = 8203520
kernel.msgmni = 10240
kernel.msgmax = 18192
net.ipv4.tcp_mem  = 379008        505344        758016
net.ipv4.tcp_wmem = 4096        16384        4194304
net.ipv4.tcp_rmem = 4096        87380        4194304
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.unix.max_dgram_qlen = 8024 
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_syn_backlog=81920
net.ipv4.tcp_synack_retries=3
net.ipv4.tcp_syn_retries=3
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 20000 65000
net.ipv4.tcp_max_tw_buckets = 200000
net.ipv4.route.max_size = 5242880
net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_timestamps=0
net.core.somaxconn=12590 
kernel.core_pattern = /core_data/core-%e-%p-%t
vm.overcommit_memory=1

[NathanFreeman]

看看服务器或者redis服务器的ulimit -n是多少

[NathanFreeman]

检查一下redis客户端的最大连接数，这里段错误是因为连接超时，抛异常了没有捕获

[mdys]

ulimit -n
1024000

[mdys]

• soft nofile 1024000
• hard nofile 1024000
  root soft nofile 1024000
  root hard nofile 1024000
• soft nproc unlimited
• hard nproc unlimited

[mdys]

redis配置也没问题 ，就是高并发时候 会出现链接问题。 目前我用的 Table 替代的 redis，但有些功能Table不能满足. 所以还是希望能redis.
我的场景是websocket 收发二进制声音媒体文件，二进制文件比较大一直发送的话 会一直触发 onopen onmessage，然后redis就撑不住了

[NathanFreeman]

执行redis-cli，输入config get maxclients看看是多少
或者在实例化完$redis_pool之后，执行$redis_pool->fill()提前创建redis

[NathanFreeman]

你这个10万个连接有点多了，控制一下进程可以创建的redis数量，最好所有进程可以创建的redis连接数加起来小于redis允许的客户端最大连接数

[mdys]

执行redis-cli，输入config get maxclients看看是多少 或者在实例化完$redis_pool之后，执行$redis_pool->fill()提前创建redis

OK 我试试提前创建

[matyhtf]

可能是连接过多，导致 connect 累积的时间过长超过了 1秒的超时限制，建议降低连接数量，将超时时间调大。解决此 fatal error