protocol/server: take conf->mutex in gf_server_check_setxattr_cmd - #4739
Open
ThalesBarretto wants to merge 1 commit into
Open
protocol/server: take conf->mutex in gf_server_check_setxattr_cmd#4739ThalesBarretto wants to merge 1 commit into
ThalesBarretto wants to merge 1 commit into
Conversation
gf_server_check_setxattr_cmd() walks conf->xprt_list with list_for_each_entry() without holding conf->mutex. Every other accessor of the list takes the lock, including the disconnect path in server_rpc_notify(), which list_del_init()s the transport under conf->mutex and then drops its last reference, freeing it. The unlocked walk can therefore dereference a node that a concurrent disconnect has just unlinked and freed: either a SIGSEGV (the freed node's list.next is reused/zeroed, list_entry() yields a wild pointer and the walk faults on xprt->total_bytes_read) or an infinite loop (list_del_init() self-links the node, so the walk never terminates and the request-handler thread is lost at 100% CPU). Both were reproduced on 11.2; the loop reproduces under plain load in a few minutes on a default-config brick. The lock was omitted when the walk was added in f068f6e ("stat enhancements", 2010), when no xprt_list site was locked. The convention was introduced by 910925e ("protocol/server: add and remove the transports from the list, inside the lock", 2012, BZ 803815 - the same crash), which locked the sibling gf_server_check_getxattr_cmd() but missed this one. Take conf->mutex around the walk, mirroring gf_server_check_getxattr_cmd(). gf_smsg() stays outside the critical section as it reads only the local accumulators. The caller holds no lock, so this cannot self-deadlock. Fixes: gluster#4639 Signed-off-by: Thales Antunes de Oliveira Barretto <thales.barretto.git@gmail.com>
Contributor
Author
|
@jiankyu your review would be very much appreciated here |
amarts
approved these changes
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gf_server_check_setxattr_cmd()walksconf->xprt_listwithlist_for_each_entry()without holding
conf->mutex. Every other accessor of that list takes the lock — includingthe disconnect path in
server_rpc_notify(), whichlist_del_init()s the transport underconf->mutexand then drops its last reference, freeing it. The unlocked walk can thereforeland on a node that a concurrent disconnect has just unlinked and freed. Two outcomes, both
reproduced on 11.2:
memory has been reused, its
list.nextis no longer a valid list pointer,list_entry()yields a wild
xprt, and the walk faults readingxprt->total_bytes_read. This matches thecrash in glusterfsd crash caused by unprotected list traversal in gf_server_check_setxattr_cmd() #4639 (
frame: op(SETXATTR),signal received: 11).list_del_init()self-links the removed node, so if the reader is onthat node the walk never terminates: the rpcsvc request-handler thread spins at 100% CPU and
is lost. This reproduces under plain load (one client issuing
setfattr -n trusted.io-stats-dumpwhile others connect/disconnect) within minutes on adefault-config brick.
The reader runs on the rpcsvc request-handler pool and the disconnect on the epoll thread, so
the race does not depend on
event-threads.Why the lock is missing. The walk was added in f068f6e ("stat enhancements", 2010)
when no
xprt_listsite was locked. The locking convention was introduced by 910925e("protocol/server: add and remove the transports from the list, inside the lock", 2012,
BZ 803815 — the same statedump-vs-disconnect crash). That change locked the sibling
gf_server_check_getxattr_cmd()but missed this one, 30 lines below it in the same file.#4639 is the second report of the same class.
Fix. Take
conf->mutexaround the walk, mirroringgf_server_check_getxattr_cmd().gf_smsg()stays outside the critical section — it reads only the two local accumulators,so the critical section is exactly the loop. The caller (
server4_0_setxattr()) holds nolock and nothing inside the loop acquires one, so this cannot self-deadlock.
Verification. Built and A/B'd: unpatched, the walk runs with
conf->mutexunheld;patched, the reader owns
conf->mutexfor the whole walk, and the disconnect path — whichtakes the same lock before
list_del_init()— serializes behind it instead of unlinking anode mid-walk.
Backport. The affected function is byte-identical from v10.5 through
devel;git apply --checkconfirms the patch applies cleanly torelease-10andrelease-11.Backport PRs to follow once this merges.
Fixes: #4639