Thanks! Both applied.
--b.
On Fri, Nov 10, 2017 at 10:19:16AM +0300, Vasily Averin wrote:
> lockd and nfsd inet[6]addr notifiers use pointer that can be changed during execution.
>
> lockd_inetaddr_event()
> ...
> if (nlmsvc_rqst) {
> ...
> svc_age_temp_xprts_now(nlmsvc_rqst->rq_server, (struct sockaddr *)&sin);
> }
>
> Usually access to nlmsvc_rqst is protected by nlmsvc_mutex
> However lockd_inet[6]addr_event does not take the mutex,
> therefore nlmsvc_rqst can be changed during execution.
>
> As result "if (nlmsvc_rqst)" can be passed,
> then another thread frees the memory or zeroes this pointer,
> and then svc_age_temp_xprts_now crash the host on access to already freed memory.
>
> Moreover on initialization nlmsvc_rqst can be temporally set to ERR_PTR.
>
> NFSD have similar issue, its reproducer is below
>
> 1) start nfsd on host
> # service nfs start
>
> 2) create separate net and mount namespaces:
> # unshare -m -n ; mount -t nfsd nfsd /proc/fs/nfsd
>
> 3) execute screen (we need 2 consoles with newly created namespaces)
> 4) on first console:
> # ifconfig lo up
> # while : ; do ip a a 1.2.3.4/32 dev lo ; do ip a d 1.2.3.4/32 dev lo ; done
>
> 5) on second console:
> # while : ; do echo 1 > /proc/fs/nfsd/threads ; sleep 1 ; echo 0 > /proc/fs/nfsd/threads ; sleep 1 ; done
>
> Result: crash inside nfsd_inteddr_event(), see demsg in attachment.
>
> We cannot add mutexes into notifiers because inet6addr notifiers should be atomic.
>
> To fix the problem I use atomic counter and waitqueue:
> counter allows notifier to access the pointer,
> waitqueue allows to delay stop of service until notifier is in use.
>
> Vasily Averin (2):
> race of lockd inetaddr notifiers vs nlmsvc_rqst change
> race of nfsd inetaddr notifiers vs nn->nfsd_serv change
>
> fs/lockd/svc.c | 16 ++++++++++++++--
> fs/nfsd/netns.h | 3 +++
> fs/nfsd/nfsctl.c | 3 +++
> fs/nfsd/nfssvc.c | 14 +++++++++++---
> 4 files changed, 31 insertions(+), 5 deletions(-)
>
> --
> 2.7.4
>
> [ 604.294055] nfsd_inetaddr_event: removed 1.2.3.4
> [ 604.294060] nfsd: last server has exited, flushing export cache
> [ 604.295922] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
> [ 604.296189] IP: _raw_spin_lock_bh+0x1b/0x30
> [ 604.296189] PGD 5a596067 P4D 5a596067 PUD 3052e067 PMD 0
> [ 604.296189] Oops: 0002 [#1] SMP
> [ 604.298844] Modules linked in: binfmt_misc nfsd auth_rpcgss nfs_acl lockd(E) grace ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw iptable_security ebtable_filter ebtables ip6table_filter ip6_tables sunrpc joydev ppdev virtio_balloon crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pvpanic parport_pc pcspkr parport i2c_piix4 xfs libcrc32c virtio_console virtio_net virtio_scsi bochs_drm drm_kms_helper crc32c_intel ttm drm serio_raw virtio_pci virtio_ring virtio ata_generic pata_acpi floppy
> [ 604.302188] CPU: 6 PID: 4310 Comm: ip Tainted: G E 4.14.0-rc6+ #2
> [ 604.302188] Hardware name: Virtuozzo KVM, BIOS 1.9.1-5.3.2.vz7.7 04/01/2014
> [ 604.305117] task: ffff8e9eda512840 task.stack: ffffb1074f288000
> [ 604.305166] RIP: 0010:_raw_spin_lock_bh+0x1b/0x30
> [ 604.306034] RSP: 0018:ffffb1074f28b950 EFLAGS: 00010246
> [ 604.306034] RAX: 0000000000000000 RBX: 0000000000000038 RCX: 0000000000000000
> [ 604.307034] RDX: 0000000000000001 RSI: ffffb1074f28b9d0 RDI: 0000000000000010
> [ 604.307034] RBP: ffffb1074f28b950 R08: 00000000000190bd R09: 0000000000000000
> [ 604.307034] R10: 00000000ff000000 R11: 00000000ffffffff R12: ffffb1074f28b978
> [ 604.307034] R13: ffffb1074f28b9d0 R14: ffff8e9eefcd8ae8 R15: 0000000000000000
> [ 604.307034] FS: 00007f16f5e720c0(0000) GS:ffff8e9effd80000(0000) knlGS:0000000000000000
> [ 604.313236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 604.313236] CR2: 0000000000000010 CR3: 000000005a695005 CR4: 00000000001606e0
> [ 604.313236] Call Trace:
> [ 604.313236] svc_age_temp_xprts_now+0x4b/0x200 [sunrpc]
> [ 604.315173] nfsd_inetaddr_event+0x87/0xb0 [nfsd]
> [ 604.315173] notifier_call_chain+0x4a/0x70
> [ 604.315173] blocking_notifier_call_chain+0x43/0x60
> [ 604.315173] __inet_del_ifa+0x16b/0x2c0
> [ 604.315173] inet_rtm_deladdr+0x129/0x1c0
> [ 604.315173] rtnetlink_rcv_msg+0x1f9/0x280
> [ 604.315173] ? rtnl_calcit.isra.24+0x110/0x110
> [ 604.315173] netlink_rcv_skb+0x91/0x130
> [ 604.322850] rtnetlink_rcv+0x15/0x20
> [ 604.322850] netlink_unicast+0x18e/0x220
> [ 604.322850] netlink_sendmsg+0x2c5/0x3c0
> [ 604.325114] sock_sendmsg+0x38/0x50
> [ 604.325150] ___sys_sendmsg+0x29a/0x2f0
> [ 604.325150] ? lru_cache_add+0x3a/0x80
> [ 604.325150] ? lru_cache_add_active_or_unevictable+0x4c/0xf0
> [ 604.325150] ? __handle_mm_fault+0x9be/0x11a0
> [ 604.325150] ? handle_mm_fault+0xb1/0x200
> [ 604.325150] __sys_sendmsg+0x54/0x90
> [ 604.325150] ? __sys_sendmsg+0x54/0x90
> [ 604.325150] SyS_sendmsg+0x12/0x20
> [ 604.325150] entry_SYSCALL_64_fastpath+0x1a/0xa5
> [ 604.325150] RIP: 0033:0x7f16f5579e57
> [ 604.331665] RSP: 002b:00007fffa38b4628 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
> [ 604.332366] RAX: ffffffffffffffda RBX: 00000000006714c0 RCX: 00007f16f5579e57
> [ 604.332920] RDX: 0000000000000000 RSI: 00007fffa38b4670 RDI: 0000000000000003
> [ 604.333191] RBP: 00007fffa38bcaf0 R08: 0000000000000001 R09: fefefeff77686d74
> [ 604.333191] R10: 0000000000000006 R11: 0000000000000246 R12: 00007fffa38bc800
> [ 604.333191] R13: 0000000000000000 R14: 00007fffa38bc7a0 R15: 00007fffa38bc7a8
> [ 604.333191] Code: 00 5d c3 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 65 81 05 af 47 76 64 00 02 00 00 48 89 e5 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 02 5d c3 89 c6 e8 d4 ac 84 ff 5d c3 66 90
> [ 604.335102] RIP: _raw_spin_lock_bh+0x1b/0x30 RSP: ffffb1074f28b950
> [ 604.335102] CR2: 0000000000000010
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
