Process xxx (sshfs) of user 0 killed by SIGABRT - dumping core

[mayelle02]

Hi team,

I figured out an issue today. Since os migration to almalinux 8.9, the sshfs process killed after few minutes and the mountpoints aren't unavailable. In production, it's very terrible because, many clients are concerned.

In below, you'll see the report from abrt tool:

reason: sshfs killed by SIGABRT time: dim. 21 juil. 2024 21:14:21 CEST cmdline: sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,allow_other [user]@server01:/var/ftp /mnt uid: 0 Directory: /var/spool/abrt/ccpp-2024-07-21-21:14:21-14228 abrt_version: 2.10.9 analyzer: abrt-ccpp architecture: x86_64 executable: /usr/bin/sshfs global_pid: 14228 hostname: server01 kernel: 4.18.0-513.24.1.el8_9.x86_64 last_occurrence: 1721589261 os_release: AlmaLinux release 8.9 (Midnight Oncilla) pid: 14228 pwd: / tid: 852300 type: CCpp coredump: Binary file, 2442788864 bytes environ: Text file, 4868 bytes maps: Text file, 13514 bytes mountinfo: Text file, 20871 bytes cgroup: :12:freezer:/ :11:perf_event:/ :10:rdma:/ :9:memory:/ :8:cpuset:/ :7:net_cls,net_prio:/ :6:pids:/ :5:hugetlb:/ :4:cpu,cpuacct:/ :3:blkio:/ :2:devices:/ :1:name=systemd:/ limits: :Limit Soft Limit Hard Limit Units
:Max cpu time unlimited unlimited seconds
:Max file size unlimited unlimited bytes
:Max data size unlimited unlimited bytes
:Max stack size 8388608 unlimited bytes
:Max core file size 0 0 bytes
:Max resident set unlimited unlimited bytes
:Max processes 512881 512881 processes
:Max open files 1024 262144 files
:Max locked memory 65536 65536 bytes
:Max address space unlimited unlimited bytes
:Max file locks unlimited unlimited locks
:Max pending signals 512881 512881 signals
:Max msgqueue size 819200 819200 bytes
:Max nice priority 0 0
:Max realtime priority 0 0
:Max realtime timeout unlimited unlimited us

namespaces:
:ipc : default
:mnt : default
:net : default
:pid : default
:uts : default
:user : default
:cgroup : default
:pid_for_children : default

open_fds:
:0:/dev/null
:pos: 0
🎏 0100002
:mnt_id: 24
:
:1:/dev/null
:pos: 0
🎏 0100002
:mnt_id: 24
:
:2:/dev/null
:pos: 0
🎏 0100002
:mnt_id: 24
:
:3:/dev/fuse
:pos: 0
🎏 02100002
:mnt_id: 24
:
:4:socket:[104258]
:pos: 0
🎏 02
:mnt_id: 10

os_info:
:NAME="AlmaLinux"
:VERSION="8.9 (Midnight Oncilla)"
:ID="almalinux"
:ID_LIKE="rhel centos fedora"
:VERSION_ID="8.9"
:PLATFORM_ID="platform:el8"
:PRETTY_NAME="AlmaLinux 8.9 (Midnight Oncilla)"
:ANSI_COLOR="0;34"
:LOGO="fedora-logo-icon"
:CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
:HOME_URL="https://almalinux.org/"
:DOCUMENTATION_URL="https://wiki.almalinux.org/"
:BUG_REPORT_URL="https://bugs.almalinux.org/"
:
:ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
:ALMALINUX_MANTISBT_PROJECT_VERSION="8.9"
:REDHAT_SUPPORT_PRODUCT="AlmaLinux"
:REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

proc_pid_status:
:Name: sshfs
:Umask: 0027
:State: D (disk sleep)
:Tgid: 14228
:Ngid: 0
:Pid: 14228
:PPid: 1
:TracerPid: 0
:Uid: 0 0 0 0
:Gid: 0 0 0 0
:FDSize: 64
:Groups: 0
:NStgid: 14228
:NSpid: 14228
:NSpgid: 14228
:NSsid: 14228
:VmPeak: 3584572 kB
:VmSize: 3387968 kB
:VmLck: 0 kB
:VmPin: 0 kB
:VmHWM: 2724564 kB
:VmRSS: 2628412 kB
:RssAnon: 2626452 kB
:RssFile: 1960 kB
:RssShmem: 0 kB
:VmData: 2782796 kB
:VmStk: 136 kB
:VmExe: 76 kB
:VmLib: 9452 kB
:VmPTE: 5376 kB
:VmSwap: 0 kB
:HugetlbPages: 0 kB
:CoreDumping: 1
:THP_enabled: 1
:Threads: 12
:SigQ: 5/512881
:SigPnd: 0000000000000000
:ShdPnd: 0000000000000000
:SigBlk: 0000000000000000
:SigIgn: 0000000000001000
:SigCgt: 0000000180004003
:CapInh: 0000000000000000
:CapPrm: 000001ffffffffff
:CapEff: 000001ffffffffff
:CapBnd: 000001ffffffffff
:CapAmb: 0000000000000000
:NoNewPrivs: 0
:Seccomp: 0
:Speculation_Store_Bypass: thread vulnerable
:Cpus_allowed: ffff,ffffffff
:Cpus_allowed_list: 0-47
:Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
:Mems_allowed_list: 0-1
:voluntary_ctxt_switches: 2
:nonvoluntary_ctxt_switches: 0

Could you help me please?

Regards,

[h4sh5]

you probably need to use the coredump to get a backtrace of what the program was doing when SIGABRT happened.

don't upload the entire coredump here since there might be sensitive data in memory - but you can run gdb on the coredump like this to get a backtrace:

gdb /usr/bin/sshfs /path/to/coredump

then in the gdb prompt, type bt and paste the output here.

[mayelle02]

Hi there,

Here the bt result:
(gdb) bt
#0 (memory address) in ?? ()
Backtrace stopped: Cannot access memory at address (memory address)

That's it

I don't know if this helpful?

[mayelle02]

Hello everyone,

I got news about this issue.
I got 2 anothers crashes of sshfs process, reasons in below:

• reason: sftp_request_wait(): sshfs killed by SIGABRT
• reason: request_free(): sshfs killed by SIGABRT

Is it possible that come from abrt packages?

Let me know if you need futher information.

[h4sh5]

@mayelle02 the abrt package is the "Automatic Bug Reporting Tool", not the same as SIGABRT signal which is the Abort Signal when a crash happens (for example, when sshfs manually calls abort(), which happens in 8 different instances in sshfs.c

can you show me your output of sshfs --version ?