On 2017-11-09 11:51 AM, Patrick McLean wrote:
> On 2017-11-09 11:37 AM, Al Viro wrote:
>> On Wed, Nov 08, 2017 at 06:40:22PM -0800, Linus Torvalds wrote:
>>
>>>> Here is the BUG we are getting:
>>>>> [ 58.962528] BUG: unable to handle kernel NULL pointer dereference at 0000000000000230
>>>>> [ 58.963918] IP: vfs_statfs+0x73/0xb0
>>>
>>> The code disassembles to
>>
>>> 2a:* 48 8b b7 30 02 00 00 mov 0x230(%rdi),%rsi <-- trapping instruction
>>
>>> that matters (and that traps) but I'm almost certain that it's the
>>> "mnt->mnt_sb->s_flags" loading that is part of calculate_f_flags()
>>> when it then does
>>>
>>> flags_by_sb(mnt->mnt_sb->s_flags);
>>>
>>> and I think mnt->mnt_sb is NULL. We know it's not 'mnt' itself that is
>>> NULL, because we wouldn't have gotten this far if it was.
>>>
>>
>> All instances of struct dentry are created by __d_alloc()[*], which assigns
>> ->d_sb (never to be modified afterwards) *and* dereferences the pointer
>> it has stored in ->d_sb before the created struct dentry becomes visible
>> to anyone else. No struct dentry should ever be observed with NULL ->d_sb;
>> the only way to get that is memory corruption or looking at freed instance
>> after its memory has been reused for something else and zeroed.
>>
>> In other words, we should never observe a struct mount with NULL ->mnt.mnt_sb -
>> not without memory corruption or looking at freed instance.
>>
>> The pointer in that case should've come from exp->ex_path.mnt, exp being
>> the argument of nfsd4_encode_fattr(). Sure, it might have been a dangling
>> reference. However, it looks a lot more like a memory corruptor *OR*
>> miscompiled kernel.
>>
>> What kind of load do the reproducer boxen have and how fast does that
>> bug trigger? Would it be possible to slap something like
>> if (unlikely(!exp->exp_path.mnt->mnt_sb)) {
>> struct mount *m = real_mount(exp->exp_path.mnt);
>> printk(KERN_ERR "mnt: %p\n", exp->exp_path.mnt);
>> printk(KERN_ERR "name: [%s]\n", m->mnt_devname);
>> printk(KERN_ERR "ns: [%p]\n", m->mnt_ns);
>> printk(KERN_ERR "parent: [%p]\n", m->mnt_parent);
>> WARN_ON(1);
>> err = -EINVAL;
>> goto out_nfserr;
>> }
>> in the beginning of nfsd4_encode_fattr() (with include of ../mount.h added
>> in fs/nfsd/nfs4xdr.c) and see what will it catch?
>>
>> Both with and without randomized structs, if possible - I might be barking
>> at the wrong tree, but IMO the very first step in localizing that crap is
>> to find out whether it's toolchain-related or not.
>
That condition did not seem to trigger, and I am getting a slightly
different crash message (GPF rather than null pointer dereference). Here
is the dump from the latest crash (with CONFIG_GCC_PLUGIN_STRUCTLEAK,
CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL and CONFIG_GCC_PLUGIN_RANDSTRUCT
all enabled).
> [ 36.834232] general protection fault: 0000 [#1] SMP
> [ 36.835168] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_multiport xt_addrtype iptable_mangle iptable_raw iptable_nat nf_nat_ipv4 nf_nat gkuart(O) usbserial x86_pkg_temp_thermal ie31200_edac tpm_tis ipmi_ssif tpm_tis_core ext4 mbcache jbd2 e1000e crc32c_intel
> [ 36.839120] CPU: 1 PID: 3969 Comm: nfsd Tainted: G O 4.14.0-rc8-git-kratos-1-00053-gd93d4ce103fd-dirty #1
> [ 36.840883] Hardware name: TYAN S5510/S5510, BIOS V2.02 03/12/2013
> [ 36.841892] task: ffff88040a0b1c80 task.stack: ffffc900027bc000
> [ 36.842887] RIP: 0010:vfs_statfs+0x73/0xb0
> [ 36.843728] RSP: 0018:ffffc900027bfb30 EFLAGS: 00010202
> [ 36.844687] RAX: 0000000000000000 RBX: ffffc900027bfbf8 RCX: 000000000000180d
> [ 36.845891] RDX: 000000000000080d RSI: 0000000000000020 RDI: e2006d6574737973
> [ 36.847075] RBP: ffffc900027bfbc8 R08: 0000000000000000 R09: 00000000000000ff
> [ 36.848175] R10: 000000000038be3a R11: ffff88040b687578 R12: 0000000000000000
> [ 36.849260] R13: ffff88040d7dc400 R14: ffff88040d38b000 R15: ffffc900027bfbf8
> [ 36.850347] FS: 0000000000000000(0000) GS:ffff88041fc40000(0000) knlGS:0000000000000000
> [ 36.851891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 36.852873] CR2: 00007f049228edc0 CR3: 0000000001e0a004 CR4: 00000000001606e0
> [ 36.853942] Call Trace:
> [ 36.854667] nfsd4_encode_fattr+0x34e/0x23b0
> [ 36.855578] ? ext4_get_acl+0x1b2/0x260 [ext4]
> [ 36.856485] ? get_acl+0x7a/0xf0
> [ 36.857266] ? generic_permission+0x125/0x1a0
> [ 36.858150] nfsd4_encode_getattr+0x25/0x30
> [ 36.859002] nfsd4_encode_operation+0x98/0x1a0
> [ 36.859889] nfsd4_proc_compound+0x3eb/0x5c0
> [ 36.860736] nfsd_dispatch+0xa8/0x230
> [ 36.861538] svc_process_common+0x347/0x640
> [ 36.862383] svc_process+0x100/0x1b0
> [ 36.863204] nfsd+0xe0/0x150
> [ 36.863984] kthread+0xfc/0x130
> [ 36.864781] ? nfsd_destroy+0x50/0x50
> [ 36.865624] ? kthread_create_on_node+0x40/0x40
> [ 36.866529] ? do_group_exit+0x3a/0xb0
> [ 36.867362] ret_from_fork+0x25/0x30
> [ 36.868188] Code: d1 83 c9 08 40 f6 c6 04 0f 45 d1 89 d1 80 cd 04 40 f6 c6 08 0f 45 d1 89 d1 80 cd 08 40 f6 c6 10 0f 45 d1 89 d1 80 cd 10 83 e6 20 <48> 8b b7 b0 05 00 00 0f 45 d1 83 ca 20 89 f1 83 e1 10 89 cf 83
> [ 36.871101] RIP: vfs_statfs+0x73/0xb0 RSP: ffffc900027bfb30
> [ 36.872059] ---[ end trace 603ac898c4e2d616 ]---
I haven't been able to reproduce it with CONFIG_GCC_PLUGIN_RANDSTRUCT
disabled, so it seems like it must be a bug there. It's odd that it just
surfaced recently though, we have been using that since it was added.
> The reproducer boxen are not under particularly heavy load, they are
> serving NFS to 1 or 2 clients (which are essentially embedded devices).
> When the bug triggers, it usually triggers pretty fast and reliably, but
> it seems to only trigger on some subset of bootups. Once it fails to
> trigger, we seem to have to reboot to get it to trigger.
>
> I should be able to have some results with that added in a few hours.
> It's weirdly unreliable to reproduce this.
>
> We do have CONFIG_GCC_PLUGIN_STRUCTLEAK and
> CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL enabled on these boxes as well as
> CONFIG_GCC_PLUGIN_RANDSTRUCT as you pointed out before.
>