On 4/19/13 8:18 AM, 符永涛 wrote: > Dear Eric, > If it's racing issue where the lock is introduced? I want to study the code from you. Thank you. > essentially: xfs_remove() { ... xfs_lock_two_inodes(dp, ip, XFS_ILOCK_EXCL); ... xfs_droplink() You are 100% sure that you were running the 279.19.1 kernel? (I'm not very familiar with Oracle's clone of RHEL - I assume that they have copied all of Red Hat's work verbatim, but I have not looked) Can you verify that in: __rwsem_do_wake() the undo target looks like: out: return sem; /* undo the change to the active count, but check for a transition * 1->0 */ undo: if (rwsem_atomic_update(-RWSEM_ACTIVE_BIAS, sem) & RWSEM_ACTIVE_MASK) goto out; goto try_again; thanks, -Eric > 2013/4/19 符永涛 <yongtaofu@xxxxxxxxx <mailto:yongtaofu@xxxxxxxxx>> > > Sure the serious thing here is that it corrupt the unlinked list. The inode 0x1bd33 which trigger xfs shutdown is not 0x6b133. > > > 2013/4/19 Eric Sandeen <sandeen@xxxxxxxxxxx <mailto:sandeen@xxxxxxxxxxx>> > > On 4/19/13 4:41 AM, 符永涛 wrote: > > Dear Brian and Eric, > > > > kernel kernel-2.6.32-279.19.1.el6.x86_64.rpm <http://mirror.linux.duke.edu/pub/centos/6.3/updates/x86_64/Packages/kernel-2.6.32-279.19.1.el6.x86_64.rpm> still have this problem > > I build the kernel from this srpm > > https://oss.oracle.com/ol6/SRPMS-updates/kernel-2.6.32-279.19.1.el6.src.rpm > > > > today the shutdown happens again during test. > > Seelogs bellow: > > > > /var/log/message > > Apr 19 16:40:05 10 kernel: XFS (sdb): xfs_iunlink_remove: xfs_inotobp() returned error 22. > > Apr 19 16:40:05 10 kernel: XFS (sdb): xfs_inactive: xfs_ifree returned error 22 > > Apr 19 16:40:05 10 kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called from line 1184 of file fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa02d4bda > > Apr 19 16:40:05 10 kernel: XFS (sdb): I/O Error Detected. Shutting down filesystem > > Apr 19 16:40:05 10 kernel: XFS (sdb): Please umount the filesystem and rectify the problem(s) > > Apr 19 16:40:07 10 kernel: XFS (sdb): xfs_log_force: error 5 returned. > > Apr 19 16:40:37 10 kernel: XFS (sdb): xfs_log_force: error 5 returned. > > > > systemtap script output: > > --- xfs_imap -- module("xfs").function("xfs_imap@fs/xfs/xfs_ialloc.c:1257").return -- return=0x16 > > vars: mp=0xffff88101801e800 tp=0xffff880ff143ac70 ino=0xffffffff imap=0xffff88100e93bc08 flags=0x0 agbno=? agino=? agno=? blks_per_cluster=? chunk_agbno=? cluster_agbno=? error=? offset=? offset_agbno=? __func__=[...] > > mp: m_agno_log = 0x5, m_agino_log = 0x20 > > mp->m_sb: sb_agcount = 0x1c, sb_agblocks = 0xffffff0, sb_inopblog = 0x4, sb_agblklog = 0x1c, sb_dblocks = 0x1b4900000 > > imap: im_blkno = 0x0, im_len = 0xe778, im_boffset = 0xd997 > > kernel backtrace: > > Returning from: 0xffffffffa02b4260 : xfs_imap+0x0/0x280 [xfs] > > Returning to : 0xffffffffa02b9d59 : xfs_inotobp+0x49/0xc0 [xfs] > > 0xffffffffa02b9ec1 : xfs_iunlink_remove+0xf1/0x360 [xfs] > > 0xffffffff814ede89 > > 0x0 (inexact) > > user backtrace: > > 0x3ec260e5ad [/lib64/libpthread-2.12.so <http://libpthread-2.12.so> <http://libpthread-2.12.so>+0xe5ad/0x219000] > > > > --- xfs_iunlink_remove -- module("xfs").function("xfs_iunlink_remove@fs/xfs/xfs_inode.c:1681").return -- return=0x16 > > vars: tp=0xffff880ff143ac70 ip=0xffff8811ed111000 next_ino=? mp=? agi=? dip=? agibp=? ibp=? agno=? agino=? next_agino=? last_ibp=? last_dip=0xffff881000000001 bucket_index=? offset=? last_offset=0xffffffffffff8811 error=? __func__=[...] > > ip: i_ino = 0x1bd33, i_flags = 0x0 > > ip->i_d: di_nlink = 0x0, di_gen = 0x53068791 > > > > debugfs events trace: > > https://docs.google.com/file/d/0B7n2C4T5tfNCREZtdC1yamc0RnM/edit?usp=sharing > > Same issue, one file was unlinked twice in a race: > > === ino 0x6b133 === > <...>-4477 [003] 2721.176790: xfs_iunlink: dev 8:16 ino 0x6b133 > <...>-4477 [003] 2721.176839: xfs_iunlink_remove: dev 8:16 ino 0x6b133 > <...>-4477 [009] 3320.127227: xfs_iunlink: dev 8:16 ino 0x6b133 > <...>-4477 [001] 3320.141126: xfs_iunlink_remove: dev 8:16 ino 0x6b133 > <...>-4477 [003] 7973.136368: xfs_iunlink: dev 8:16 ino 0x6b133 > <...>-4479 [018] 7973.158457: xfs_iunlink: dev 8:16 ino 0x6b133 > <...>-4479 [018] 7973.158497: xfs_iunlink_remove: dev 8:16 ino 0x6b133 > > -Eric > > > > > -- > 符永涛 > > > > > -- > 符永涛 _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs