On Thu, 2024-10-24 at 21:40 +0800, liujian (CE) wrote:
>
>
> 在 2024/10/24 20:57, Trond Myklebust 写道:
> > On Thu, 2024-10-24 at 02:20 +0000, Trond Myklebust wrote:
> > > On Thu, 2024-10-24 at 09:55 +0800, Liu Jian wrote:
> > > > BUG: KASAN: slab-use-after-free in
> > > > tcp_write_timer_handler+0x156/0x3e0
> > > > Read of size 1 at addr ffff888111f322cd by task swapper/0/0
> > > >
> > > > CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.0-rc4-
> > > > dirty
> > > > #7
> > > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > > > 1.15.0-
> > > > 1
> > > > Call Trace:
> > > > <IRQ>
> > > > dump_stack_lvl+0x68/0xa0
> > > > print_address_description.constprop.0+0x2c/0x3d0
> > > > print_report+0xb4/0x270
> > > > kasan_report+0xbd/0xf0
> > > > tcp_write_timer_handler+0x156/0x3e0
> > > > tcp_write_timer+0x66/0x170
> > > > call_timer_fn+0xfb/0x1d0
> > > > __run_timers+0x3f8/0x480
> > > > run_timer_softirq+0x9b/0x100
> > > > handle_softirqs+0x153/0x390
> > > > __irq_exit_rcu+0x103/0x120
> > > > irq_exit_rcu+0xe/0x20
> > > > sysvec_apic_timer_interrupt+0x76/0x90
> > > > </IRQ>
> > > > <TASK>
> > > > asm_sysvec_apic_timer_interrupt+0x1a/0x20
> > > > RIP: 0010:default_idle+0xf/0x20
> > > > Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90
> > > > 90
> > > > 90
> > > > 90 90
> > > > 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d 33 f8 25 00 fb f4 <fa>
> > > > c3
> > > > cc
> > > > cc cc
> > > > cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
> > > > RSP: 0018:ffffffffa2007e28 EFLAGS: 00000242
> > > > RAX: 00000000000f3b31 RBX: 1ffffffff4400fc7 RCX:
> > > > ffffffffa09c3196
> > > > RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> > > > ffffffff9f00590f
> > > > RBP: 0000000000000000 R08: 0000000000000001 R09:
> > > > ffffed102360835d
> > > > R10: ffff88811b041aeb R11: 0000000000000001 R12:
> > > > 0000000000000000
> > > > R13: ffffffffa202d7c0 R14: 0000000000000000 R15:
> > > > 00000000000147d0
> > > > default_idle_call+0x6b/0xa0
> > > > cpuidle_idle_call+0x1af/0x1f0
> > > > do_idle+0xbc/0x130
> > > > cpu_startup_entry+0x33/0x40
> > > > rest_init+0x11f/0x210
> > > > start_kernel+0x39a/0x420
> > > > x86_64_start_reservations+0x18/0x30
> > > > x86_64_start_kernel+0x97/0xa0
> > > > common_startup_64+0x13e/0x141
> > > > </TASK>
> > > >
> > > > Allocated by task 595:
> > > > kasan_save_stack+0x24/0x50
> > > > kasan_save_track+0x14/0x30
> > > > __kasan_slab_alloc+0x87/0x90
> > > > kmem_cache_alloc_noprof+0x12b/0x3f0
> > > > copy_net_ns+0x94/0x380
> > > > create_new_namespaces+0x24c/0x500
> > > > unshare_nsproxy_namespaces+0x75/0xf0
> > > > ksys_unshare+0x24e/0x4f0
> > > > __x64_sys_unshare+0x1f/0x30
> > > > do_syscall_64+0x70/0x180
> > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > > >
> > > > Freed by task 100:
> > > > kasan_save_stack+0x24/0x50
> > > > kasan_save_track+0x14/0x30
> > > > kasan_save_free_info+0x3b/0x60
> > > > __kasan_slab_free+0x54/0x70
> > > > kmem_cache_free+0x156/0x5d0
> > > > cleanup_net+0x5d3/0x670
> > > > process_one_work+0x776/0xa90
> > > > worker_thread+0x2e2/0x560
> > > > kthread+0x1a8/0x1f0
> > > > ret_from_fork+0x34/0x60
> > > > ret_from_fork_asm+0x1a/0x30
> > > >
> > > > Reproduction script:
> > > >
> > > > mkdir -p /mnt/nfsshare
> > > > mkdir -p /mnt/nfs/netns_1
> > > > mkfs.ext4 /dev/sdb
> > > > mount /dev/sdb /mnt/nfsshare
> > > > systemctl restart nfs-server
> > > > chmod 777 /mnt/nfsshare
> > > > exportfs -i -o rw,no_root_squash *:/mnt/nfsshare
> > > >
> > > > ip netns add netns_1
> > > > ip link add name veth_1_peer type veth peer veth_1
> > > > ifconfig veth_1_peer 11.11.0.254 up
> > > > ip link set veth_1 netns netns_1
> > > > ip netns exec netns_1 ifconfig veth_1 11.11.0.1
> > > >
> > > > ip netns exec netns_1 /root/iptables -A OUTPUT -d 11.11.0.254 -
> > > > p
> > > > tcp
> > > > \
> > > > --tcp-flags FIN FIN -j DROP
> > > >
> > > > (note: In my environment, a DESTROY_CLIENTID operation is
> > > > always
> > > > sent
> > > > immediately, breaking the nfs tcp connection.)
> > > > ip netns exec netns_1 timeout -s 9 300 mount -t nfs -o
> > > > proto=tcp,vers=4.1 \
> > > > 11.11.0.254:/mnt/nfsshare /mnt/nfs/netns_1
> > > >
> > > > ip netns del netns_1
> > > >
> > > > The reason here is that the tcp socket in netns_1 (nfs side)
> > > > has
> > > > been
> > > > shutdown and closed (done in xs_destroy), but the FIN message
> > > > (with
> > > > ack)
> > > > is discarded, and the nfsd side keeps sending retransmission
> > > > messages.
> > > > As a result, when the tcp sock in netns_1 processes the
> > > > received
> > > > message,
> > > > it sends the message (FIN message) in the sending queue, and
> > > > the
> > > > tcp
> > > > timer
> > > > is re-established. When the network namespace is deleted, the
> > > > net
> > > > structure
> > > > accessed by tcp's timer handler function causes problems.
> > > >
> > > > The modification here aborts the TCP connection directly in
> > > > xs_destroy().
> > > >
> > > > Fixes: 26abe14379f8 ("net: Modify sk_alloc to not reference
> > > > count
> > > > the
> > > > netns of kernel sockets.")
> > > > Signed-off-by: Liu Jian <liujian56@xxxxxxxxxx>
> > > > ---
> > > > net/sunrpc/xprtsock.c | 3 +++
> > > > 1 file changed, 3 insertions(+)
> > > >
> > > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
> > > > index 0e1691316f42..91ee3484155a 100644
> > > > --- a/net/sunrpc/xprtsock.c
> > > > +++ b/net/sunrpc/xprtsock.c
> > > > @@ -1287,6 +1287,9 @@ static void xs_reset_transport(struct
> > > > sock_xprt
> > > > *transport)
> > > > release_sock(sk);
> > > > mutex_unlock(&transport->recv_mutex);
> > > >
> > > > + if (sk->sk_prot == &tcp_prot)
> > > > + tcp_abort(sk, ECONNABORTED);
> > > We've already called kernel_sock_shutdown(sock, SHUT_RDWR), and
> > > we're
> > > about to close the socket. Why on earth should we need a hack
> > > like
> > > the
> > > above in order to abort the connection at this point?
> > >
> > > This would appear to be a networking layer bug, and not an RPC
> > > level
> > > problem.
> > >
> > To put this differently: if a use after free can occur in the
> > kernel
> > when the RPC layer closes a TCP socket and then exits the network
> > namespace, then can't that occur when a userland application does
> > the
> > same?
> >
> > If not, then what prevents it from happening?
> The socket created by the userspace program obtains the reference
> counting of the namespace, but the kernel socket does not.
>
> There's some discussion here:
> https://lore.kernel.org/all/CANn89iJE5anTbyLJ0TdGAqGsE+GichY3YzQECjNUVMz=G3bcQg@xxxxxxxxxxxxxx/
OK... So then it looks to me as if NFS, SMB, AFS, and any other
networked filesystem that can be started from inside a container is
going to need to do the same thing that rds appears to be doing.
Should there perhaps be a helper function in the networking layer for
this?
> > > > +
> > > > trace_rpc_socket_close(xprt, sock);
> > > > __fput_sync(filp);
> > > >
>
>
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@xxxxxxxxxxxxxxx
