Skip to content

protocol/server: take conf->mutex in gf_server_check_setxattr_cmd - #4739

Open
ThalesBarretto wants to merge 1 commit into
gluster:develfrom
ThalesBarretto:issue-4639-server-xprt-list-lock
Open

protocol/server: take conf->mutex in gf_server_check_setxattr_cmd#4739
ThalesBarretto wants to merge 1 commit into
gluster:develfrom
ThalesBarretto:issue-4639-server-xprt-list-lock

Conversation

@ThalesBarretto

Copy link
Copy Markdown
Contributor

gf_server_check_setxattr_cmd() walks conf->xprt_list with list_for_each_entry()
without holding conf->mutex. Every other accessor of that list takes the lock — including
the disconnect path in server_rpc_notify(), which list_del_init()s the transport under
conf->mutex and then drops its last reference, freeing it. The unlocked walk can therefore
land on a node that a concurrent disconnect has just unlinked and freed. Two outcomes, both
reproduced on 11.2:

  • SIGSEGV — the walk reads a transport that has been unlinked and freed; when the freed
    memory has been reused, its list.next is no longer a valid list pointer, list_entry()
    yields a wild xprt, and the walk faults reading xprt->total_bytes_read. This matches the
    crash in glusterfsd crash caused by unprotected list traversal in gf_server_check_setxattr_cmd() #4639 (frame: op(SETXATTR), signal received: 11).
  • Infinite looplist_del_init() self-links the removed node, so if the reader is on
    that node the walk never terminates: the rpcsvc request-handler thread spins at 100% CPU and
    is lost. This reproduces under plain load (one client issuing
    setfattr -n trusted.io-stats-dump while others connect/disconnect) within minutes on a
    default-config brick.

The reader runs on the rpcsvc request-handler pool and the disconnect on the epoll thread, so
the race does not depend on event-threads.

Why the lock is missing. The walk was added in f068f6e ("stat enhancements", 2010)
when no xprt_list site was locked. The locking convention was introduced by 910925e
("protocol/server: add and remove the transports from the list, inside the lock", 2012,
BZ 803815 — the same statedump-vs-disconnect crash). That change locked the sibling
gf_server_check_getxattr_cmd() but missed this one, 30 lines below it in the same file.
#4639 is the second report of the same class.

Fix. Take conf->mutex around the walk, mirroring gf_server_check_getxattr_cmd().
gf_smsg() stays outside the critical section — it reads only the two local accumulators,
so the critical section is exactly the loop. The caller (server4_0_setxattr()) holds no
lock and nothing inside the loop acquires one, so this cannot self-deadlock.

Verification. Built and A/B'd: unpatched, the walk runs with conf->mutex unheld;
patched, the reader owns conf->mutex for the whole walk, and the disconnect path — which
takes the same lock before list_del_init() — serializes behind it instead of unlinking a
node mid-walk.

Backport. The affected function is byte-identical from v10.5 through devel;
git apply --check confirms the patch applies cleanly to release-10 and release-11.
Backport PRs to follow once this merges.

Fixes: #4639

gf_server_check_setxattr_cmd() walks conf->xprt_list with
list_for_each_entry() without holding conf->mutex. Every other accessor
of the list takes the lock, including the disconnect path in
server_rpc_notify(), which list_del_init()s the transport under
conf->mutex and then drops its last reference, freeing it. The unlocked
walk can therefore dereference a node that a concurrent disconnect has
just unlinked and freed: either a SIGSEGV (the freed node's list.next is
reused/zeroed, list_entry() yields a wild pointer and the walk faults on
xprt->total_bytes_read) or an infinite loop (list_del_init() self-links
the node, so the walk never terminates and the request-handler thread is
lost at 100% CPU). Both were reproduced on 11.2; the loop reproduces
under plain load in a few minutes on a default-config brick.

The lock was omitted when the walk was added in f068f6e ("stat
enhancements", 2010), when no xprt_list site was locked. The convention
was introduced by 910925e ("protocol/server: add and remove the
transports from the list, inside the lock", 2012, BZ 803815 - the same
crash), which locked the sibling gf_server_check_getxattr_cmd() but
missed this one.

Take conf->mutex around the walk, mirroring gf_server_check_getxattr_cmd().
gf_smsg() stays outside the critical section as it reads only the local
accumulators. The caller holds no lock, so this cannot self-deadlock.

Fixes: gluster#4639
Signed-off-by: Thales Antunes de Oliveira Barretto <thales.barretto.git@gmail.com>
@ThalesBarretto

Copy link
Copy Markdown
Contributor Author

@jiankyu your review would be very much appreciated here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

glusterfsd crash caused by unprotected list traversal in gf_server_check_setxattr_cmd()

2 participants