On 12/17/24 10:30 AM, Li Lingfeng wrote:
Hi,
after analysis, we think that this issue is not introduced by commit
2d4a532d385f ("nfsd: ensure that clp->cl_revoked list is protected by
clp->cl_lock") but by commit 83e733161fde ("nfsd: avoid race after
unhash_delegation_locked()").
Therefore, kernel versions earlier than 6.9 do not involve this issue.
A more practical question is: has anyone reproduced the reported crash
on a pre-v6.9 kernel?
I recall (dimly) that we knew that 8dd91e8d31fe ("nfsd: fix race between
laundromat and free_stateid") could not be cleanly applied before v6.9.
It was less clear at the time whether a more extensive LTS backport
would be required.
// normal case 1 -- free deleg by delegreturn
1) OP_DELEGRETURN
nfsd4_delegreturn
nfsd4_lookup_stateid
destroy_delegation
destroy_unhashed_deleg
nfs4_unlock_deleg_lease
vfs_setlease // unlock
nfs4_put_stid // put last refcount
idr_remove // remove from cl_stateids
s->sc_free // free deleg
2) OP_FREE_STATEID
nfsd4_free_stateid
find_stateid_locked // can not find the deleg in cl_stateids
// normal case 2 -- free deleg by laundromat
nfs4_laundromat
state_expired
unhash_delegation_locked // set NFS4_REVOKED_DELEG_STID
list_add // add the deleg to reaplist
list_first_entry // get the deleg from reaplist
revoke_delegation
destroy_unhashed_deleg
nfs4_unlock_deleg_lease
nfs4_put_stid
// abnormal case
nfs4_laundromat
state_expired
unhash_delegation_locked
// set NFS4_REVOKED_DELEG_STID
list_add
// add the deleg to reaplist
1) OP_DELEGRETURN
nfsd4_delegreturn
nfsd4_lookup_stateid
nfsd4_stid_check_stateid_generation
nfsd4_verify_open_stid
// check NFS4_REVOKED_DELEG_STID
// and return nfserr_deleg_revoked
// skip destroy_delegation
2) OP_FREE_STATEID
nfsd4_free_stateid
// check NFS4_REVOKED_DELEG_STID
list_del_init
// remove deleg from reaplist
nfs4_put_stid
// free deleg
list_first_entry
// cant not get the deleg from reaplist
Before commit 83e733161fde ("nfsd: avoid race after
unhash_delegation_locked()"), nfs4_laundromat --> unhash_delegation_locked
would not set NFS4_REVOKED_DELEG_STID for the deleg.
So the description "it marks the delegation stid revoked" in the CVE fix
patch does not hold true. And the OP_FREE_STATEID operation will not
release the deleg.
Thanks.
在 2024/11/6 1:10, Greg Kroah-Hartman 写道:
Description
===========
In the Linux kernel, the following vulnerability has been resolved:
nfsd: fix race between laundromat and free_stateid
There is a race between laundromat handling of revoked delegations
and a client sending free_stateid operation. Laundromat thread
finds that delegation has expired and needs to be revoked so it
marks the delegation stid revoked and it puts it on a reaper list
but then it unlock the state lock and the actual delegation revocation
happens without the lock. Once the stid is marked revoked a racing
free_stateid processing thread does the following (1) it calls
list_del_init() which removes it from the reaper list and (2) frees
the delegation stid structure. The laundromat thread ends up not
calling the revoke_delegation() function for this particular delegation
but that means it will no release the lock lease that exists on
the file.
Now, a new open for this file comes in and ends up finding that
lease list isn't empty and calls nfsd_breaker_owns_lease() which ends
up trying to derefence a freed delegation stateid. Leading to the
followint use-after-free KASAN warning:
kernel:
==================================================================
kernel: BUG: KASAN: slab-use-after-free in
nfsd_breaker_owns_lease+0x140/0x160 [nfsd]
kernel: Read of size 8 at addr ffff0000e73cd0c8 by task nfsd/6205
kernel:
kernel: CPU: 2 UID: 0 PID: 6205 Comm: nfsd Kdump: loaded Not tainted
6.11.0-rc7+ #9
kernel: Hardware name: Apple Inc. Apple Virtualization Generic
Platform, BIOS 2069.0.0.0.0 08/03/2024
kernel: Call trace:
kernel: dump_backtrace+0x98/0x120
kernel: show_stack+0x1c/0x30
kernel: dump_stack_lvl+0x80/0xe8
kernel: print_address_description.constprop.0+0x84/0x390
kernel: print_report+0xa4/0x268
kernel: kasan_report+0xb4/0xf8
kernel: __asan_report_load8_noabort+0x1c/0x28
kernel: nfsd_breaker_owns_lease+0x140/0x160 [nfsd]
kernel: nfsd_file_do_acquire+0xb3c/0x11d0 [nfsd]
kernel: nfsd_file_acquire_opened+0x84/0x110 [nfsd]
kernel: nfs4_get_vfs_file+0x634/0x958 [nfsd]
kernel: nfsd4_process_open2+0xa40/0x1a40 [nfsd]
kernel: nfsd4_open+0xa08/0xe80 [nfsd]
kernel: nfsd4_proc_compound+0xb8c/0x2130 [nfsd]
kernel: nfsd_dispatch+0x22c/0x718 [nfsd]
kernel: svc_process_common+0x8e8/0x1960 [sunrpc]
kernel: svc_process+0x3d4/0x7e0 [sunrpc]
kernel: svc_handle_xprt+0x828/0xe10 [sunrpc]
kernel: svc_recv+0x2cc/0x6a8 [sunrpc]
kernel: nfsd+0x270/0x400 [nfsd]
kernel: kthread+0x288/0x310
kernel: ret_from_fork+0x10/0x20
This patch proposes a fixed that's based on adding 2 new additional
stid's sc_status values that help coordinate between the laundromat
and other operations (nfsd4_free_stateid() and nfsd4_delegreturn()).
First to make sure, that once the stid is marked revoked, it is not
removed by the nfsd4_free_stateid(), the laundromat take a reference
on the stateid. Then, coordinating whether the stid has been put
on the cl_revoked list or we are processing FREE_STATEID and need to
make sure to remove it from the list, each check that state and act
accordingly. If laundromat has added to the cl_revoke list before
the arrival of FREE_STATEID, then nfsd4_free_stateid() knows to remove
it from the list. If nfsd4_free_stateid() finds that operations arrived
before laundromat has placed it on cl_revoke list, it marks the state
freed and then laundromat will no longer add it to the list.
Also, for nfsd4_delegreturn() when looking for the specified stid,
we need to access stid that are marked removed or freeable, it means
the laundromat has started processing it but hasn't finished and this
delegreturn needs to return nfserr_deleg_revoked and not
nfserr_bad_stateid. The latter will not trigger a FREE_STATEID and the
lack of it will leave this stid on the cl_revoked list indefinitely.
The Linux kernel CVE team has assigned CVE-2024-50106 to this issue.
Affected and fixed versions
===========================
Issue introduced in 3.17 with commit 2d4a532d385f and fixed in
6.11.6 with commit 967faa26f313
Issue introduced in 3.17 with commit 2d4a532d385f and fixed in
6.12-rc5 with commit 8dd91e8d31fe
Please see https://www.kernel.org for a full list of currently supported
kernel versions by the kernel community.
Unaffected versions might change over time as fixes are backported to
older supported kernel versions. The official CVE entry at
https://cve.org/CVERecord/?id=CVE-2024-50106
will be updated if fixes are backported, please check that for the most
up to date information about this issue.
Affected files
==============
The file(s) affected by this issue are:
fs/nfsd/nfs4state.c
fs/nfsd/state.h
Mitigation
==========
The Linux kernel CVE team recommends that you update to the latest
stable kernel version for this, and many other bugfixes. Individual
changes are never tested alone, but rather are part of a larger kernel
release. Cherry-picking individual commits is not recommended or
supported by the Linux kernel community at all. If however, updating to
the latest release is impossible, the individual changes to resolve this
issue can be found at these commits:
https://git.kernel.org/stable/
c/967faa26f313a62e7bebc55d5b8122eaee43b929
https://git.kernel.org/stable/
c/8dd91e8d31febf4d9cca3ae1bb4771d33ae7ee5a
--
Chuck Lever
