Re: [PATCH] NFSV4: fix rpc_task use-after-free when open concurrently

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





在 2024/9/28 4:58, Anna Schumaker 写道:
Hi Yang,

On 9/26/24 2:12 AM, Yang Erkun wrote:
From: Yang Erkun <yangerkun@xxxxxxxxxx>

Two threads that work with the same cred try to open different files
concurrently, they will utilize the same nfs4_state_owner. And in order
to sequential open request send to server, the second task will fall
into RPC_TASK_QUEUED in nfs_wait_on_sequence since there is already one
work doing the open operation. Furthermore, the second task will wait
until the first task completes its work, call rpc_wake_up_queued_task in
nfs_release_seqid to wake up the second task, allowing it to complete
the remaining open operation.

The preceding logic does not cause any problems under normal
circumstances. However, when once we force an unmount using `umount -f`,
the function nfs_umount_begin attempts to kill all tasks by calling
rpc_signal_task. This help wake up the second task, but it sets the
status to -ERESTARTSYS. This status prevents `nfs4_open_release` from
calling `nfs4_opendata_to_nfs4_state`. Consequently, while the second
task will be freed, the original tasks will still exist in
sequence->list(see nfs_release_seqid). Latter, when the first thread
calls nfs_release_seqid and attempts to wake up the second task, it will
trigger the uaf.

To resolve this issue, ensure rpc_task will remove it from
sequence->list by adding nfs_release_seqid in nfs4_open_release.

==================================================================
BUG: KASAN: slab-use-after-free in rpc_wake_up_queued_task+0xbb/0xc0
Read of size 8 at addr ff11000007639930 by task bash/792

CPU: 0 UID: 0 PID: 792 Comm: bash Tainted: G    B   W
6.11.0-09960-gd10b58fe53dc-dirty #10
Tainted: [B]=BAD_PAGE, [W]=WARN
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.16.1-2.fc37 04/01/2014
Call Trace:
  <TASK>
  dump_stack_lvl+0xa3/0x120
  print_address_description.constprop.0+0x63/0x510
  print_report+0xf5/0x360
  kasan_report+0xd9/0x140
  __asan_report_load8_noabort+0x24/0x40
  rpc_wake_up_queued_task+0xbb/0xc0
  nfs_release_seqid+0x1e1/0x2f0
  nfs_free_seqid+0x1a/0x40
  nfs4_opendata_free+0xc6/0x3e0
  _nfs4_do_open.isra.0+0xbe3/0x1380
  nfs4_do_open+0x28b/0x620
  nfs4_atomic_open+0x2c6/0x3a0
  nfs_atomic_open+0x4f8/0x1180
  atomic_open+0x186/0x4e0
  lookup_open.isra.0+0x3e7/0x15b0
  open_last_lookups+0x85d/0x1260
  path_openat+0x151/0x7b0
  do_filp_open+0x1e0/0x310
  do_sys_openat2+0x178/0x1f0
  do_sys_open+0xa2/0x100
  __x64_sys_openat+0xa8/0x120
  x64_sys_call+0x2507/0x4540
  do_syscall_64+0xa7/0x240
  entry_SYSCALL_64_after_hwframe+0x76/0x7e

...

Allocated by task 767:
  kasan_save_stack+0x3b/0x70
  kasan_save_track+0x1c/0x40
  kasan_save_alloc_info+0x44/0x70
  __kasan_slab_alloc+0xaf/0xc0
  kmem_cache_alloc_noprof+0x1e0/0x4f0
  rpc_new_task+0xe7/0x220
  rpc_run_task+0x27/0x7d0
  nfs4_run_open_task+0x477/0x810
  _nfs4_proc_open+0xc0/0x6d0
  _nfs4_open_and_get_state+0x178/0xc50
  _nfs4_do_open.isra.0+0x47f/0x1380
  nfs4_do_open+0x28b/0x620
  nfs4_atomic_open+0x2c6/0x3a0
  nfs_atomic_open+0x4f8/0x1180
  atomic_open+0x186/0x4e0
  lookup_open.isra.0+0x3e7/0x15b0
  open_last_lookups+0x85d/0x1260
  path_openat+0x151/0x7b0
  do_filp_open+0x1e0/0x310
  do_sys_openat2+0x178/0x1f0
  do_sys_open+0xa2/0x100
  __x64_sys_openat+0xa8/0x120
  x64_sys_call+0x2507/0x4540
  do_syscall_64+0xa7/0x240
  entry_SYSCALL_64_after_hwframe+0x76/0x7e

Freed by task 767:
  kasan_save_stack+0x3b/0x70
  kasan_save_track+0x1c/0x40
  kasan_save_free_info+0x43/0x80
  __kasan_slab_free+0x4f/0x90
  kmem_cache_free+0x199/0x4f0
  mempool_free_slab+0x1f/0x30
  mempool_free+0xdf/0x3d0
  rpc_free_task+0x12d/0x180
  rpc_final_put_task+0x10e/0x150
  rpc_do_put_task+0x63/0x80
  rpc_put_task+0x18/0x30
  nfs4_run_open_task+0x4f4/0x810
  _nfs4_proc_open+0xc0/0x6d0
  _nfs4_open_and_get_state+0x178/0xc50
  _nfs4_do_open.isra.0+0x47f/0x1380
  nfs4_do_open+0x28b/0x620
  nfs4_atomic_open+0x2c6/0x3a0
  nfs_atomic_open+0x4f8/0x1180
  atomic_open+0x186/0x4e0
  lookup_open.isra.0+0x3e7/0x15b0
  open_last_lookups+0x85d/0x1260
  path_openat+0x151/0x7b0
  do_filp_open+0x1e0/0x310
  do_sys_openat2+0x178/0x1f0
  do_sys_open+0xa2/0x100
  __x64_sys_openat+0xa8/0x120
  x64_sys_call+0x2507/0x4540
  do_syscall_64+0xa7/0x240
  entry_SYSCALL_64_after_hwframe+0x76/0x7e

Once I apply this patch I'm seeing my client hang when running xfstests generic/451 with NFS v4.0. I was wondering if you could check if you see the same hang, and please fix it if so?


I have try to reproduce this with kernel commit:

commit abf2050f51fdca0fd146388f83cddd95a57a008d
Merge: 9ab27b018649 81ee62e8d09e
Author: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
Date:   Mon Sep 23 15:27:58 2024 -0700

Merge tag 'media/v6.12-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media

And for nfs4.0/nfs4.1, all seems ok now...

Can you provide more info about the 'hang' you meet now?


Thanks,
Anna


Fixes: 24ac23ab88df ("NFSv4: Convert open() into an asynchronous RPC call")
Signed-off-by: Yang Erkun <yangerkun@xxxxxxxxxx>
---
  fs/nfs/nfs4proc.c | 1 +
  1 file changed, 1 insertion(+)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index b8ffbe52ba15..4685621ba469 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2603,6 +2603,7 @@ static void nfs4_open_release(void *calldata)
  	struct nfs4_opendata *data = calldata;
  	struct nfs4_state *state = NULL;
+ nfs_release_seqid(data->o_arg.seqid);
  	/* If this request hasn't been cancelled, do nothing */
  	if (!data->cancelled)
  		goto out_free;





[Index of Archives]     [Linux Filesystem Development]     [Linux USB Development]     [Linux Media Development]     [Video for Linux]     [Linux NILFS]     [Linux Audio Users]     [Yosemite Info]     [Linux SCSI]

  Powered by Linux