On Tue, Oct 31, 2017 at 06:51:08PM -0700, Cong Wang wrote: > On Mon, Oct 30, 2017 at 5:33 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote: > > On Mon, Oct 30, 2017 at 02:55:43PM -0700, Cong Wang wrote: > >> Hello, > >> > >> We triggered a list corruption (double add) warning below on our 4.9 > >> kernel (the 4.9 kernel we use is based on -stable release, with only a > >> few unrelated networking backports): ... > >> 4.9.34.el7.x86_64 #1 > >> Hardware name: TYAN S5512/S5512, BIOS V8.B13 03/20/2014 > >> ffffb0d48a0abb30 ffffffff8e389f47 ffffb0d48a0abb80 0000000000000000 > >> ffffb0d48a0abb70 ffffffff8e08989b 0000002400000000 ffff8d9d691e0aa0 > >> ffff8d9d7a716608 ffff8d9d691e0aa0 0000000000004000 ffff8d9d7de6d800 > >> Call Trace: > >> [<ffffffff8e389f47>] dump_stack+0x4d/0x66 > >> [<ffffffff8e08989b>] __warn+0xcb/0xf0 > >> [<ffffffff8e08991f>] warn_slowpath_fmt+0x5f/0x80 > >> [<ffffffff8e3a979c>] __list_add+0xac/0xb0 > >> [<ffffffff8e2355bb>] inode_sb_list_add+0x3b/0x50 > >> [<ffffffffc040157c>] xfs_setup_inode+0x2c/0x170 [xfs] > >> [<ffffffffc0402097>] xfs_ialloc+0x317/0x5c0 [xfs] > >> [<ffffffffc0404347>] xfs_dir_ialloc+0x77/0x220 [xfs] > > > > Inode allocation, so should be a new inode straight from the slab > > cache. THat implies memory corruption of some kind. Please turn on > > slab poisoning and try to reproduce. > > Are you sure? xfs_iget() seems searching in a cache before allocating > a new one: /me sighs You started with "I don't know the XFS code very well", so I omitted the complexity of describing about 10 different corner cases where we /could/ find the unlinked inode still in the cache via the lookup. But they aren't common cases - the common case in the real world is allocation of cache cold inodes. IOWs: "so should be a new inode straight from the slab cache". So, yes, we could find the old unlinked inode still cached in the XFS inode cache, but I don't have the time to explain how RCU lookup code works to everyone who reports a bug. All you need to understand is that all of this happens below the VFS and so inodes being reclaimed or newly allocated the in-cache inode should never, ever be on the VFS sb inode list. > >> [<ffffffff8e74cf32>] ? down_write+0x12/0x40 > >> [<ffffffffc0404972>] xfs_create+0x482/0x760 [xfs] > >> [<ffffffffc04019ae>] xfs_generic_create+0x21e/0x2c0 [xfs] > >> [<ffffffffc0401a84>] xfs_vn_mknod+0x14/0x20 [xfs] > >> [<ffffffffc0401aa6>] xfs_vn_mkdir+0x16/0x20 [xfs] > >> [<ffffffff8e226698>] vfs_mkdir+0xe8/0x140 > >> [<ffffffff8e22aa4a>] SyS_mkdir+0x7a/0xf0 > >> [<ffffffff8e74f8e0>] entry_SYSCALL_64_fastpath+0x13/0x94 > >> > >> _Without_ looking deeper, it seems this warning could be shut up by: > >> > >> --- a/fs/xfs/xfs_icache.c > >> +++ b/fs/xfs/xfs_icache.c > >> @@ -1138,6 +1138,8 @@ xfs_reclaim_inode( > >> xfs_iunlock(ip, XFS_ILOCK_EXCL); > >> > >> XFS_STATS_INC(ip->i_mount, xs_ig_reclaims); > >> + > >> + inode_sb_list_del(VFS_I(ip)); > >> > >> with properly exporting inode_sb_list_del(). Does this make any sense? > > > > No, because by this stage the inode has already been removed from > > the superblock indoe list. Doing this sort of thing here would just > > paper over whatever the underlying problem might be. > > > For me, it looks like the inode in the cache pag->pag_ici_root > is not removed from sb list before removing from cache. Sure, we have list corruption. Where we detect that corruption implies nothing about the cause of the list corruption. The two events are not connected in any way. Clearing that VFS list here does nothing to fix the problem causing the list corruption to occur. > >> Please let me know if I can provide any other information. > > > > How do you reproduce the problem? > > The warning is reported via ABRT email, we don't know what was > happening at the time of crash. Which makes it even harder to track down. Perhaps you should configure the box to crashdump on such a failure and then we can do some post-failure forensic analysis... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
