On Wed, 27 Apr 2011 16:23:07 -0700 Mark Moseley <moseleymark@xxxxxxxxx> wrote: > > I posted this to bugzilla a while back but I figured I'd paste it here too: > > ----------------------------------------------------------------------------------------------------- > > I've been getting bit by the exact same bug and been bisecting for the past > couple of weeks. It's slow going as it can sometimes take a day for the BUG() > to show up (though can also at time take 10 minutes). And I've also seen it > more than once where something was good after a day and then BUG()'d later on, > just to make things more complicated. So the upshot is that while I feel > confident enough about this latest batch of bisecting to post it here, I > wouldn't bet my life on it. I hope this isn't a case where bisecting just shows > where the bug gets exposed but not where it actually got planted :) > > Incidentally, I tried the patch from the top of this thread and it didn't seem > to make a difference. I still got bit. > > I've posted on the linux-fsdevel thread that Jeff Layton started about it, > http://www.spinics.net/lists/linux-nfs/msg20280.html if you need more details > on my setup (though I'll be happy to provide anything else you need). Though in > that thread you'll see that I'm not using autofs explicitly, the Netapp GX > cluster NFS appears to use autofs to do the implicit submounts (I'm not 100% > sure that's the correct terminology, so hopefully you know what I mean). > > Here's my bisect log, ending up at commit > e61da20a50d21725ff27571a6dff9468e4fb7146 > > git bisect start 'fs' > # good: [3c0eee3fe6a3a1c745379547c7e7c904aa64f6d5] Linux 2.6.37 > git bisect good 3c0eee3fe6a3a1c745379547c7e7c904aa64f6d5 > # bad: [c56eb8fb6dccb83d9fe62fd4dc00c834de9bc470] Linux 2.6.38-rc1 > git bisect bad c56eb8fb6dccb83d9fe62fd4dc00c834de9bc470 > # good: [7c955fca3e1d8132982148267d9efcafae849bb6] Merge branch 'for_linus' of > git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-udf-2.6 > git bisect good 7c955fca3e1d8132982148267d9efcafae849bb6 > # good: [c32b0d4b3f19c2f5d29568f8b7b72b61693f1277] fs/mpage.c: consolidate code > git bisect good c32b0d4b3f19c2f5d29568f8b7b72b61693f1277 > # bad: [f8206b925fb0eba3a11839419be118b09105d7b1] Merge branch 'for-linus' of > git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs-2.6 > git bisect bad f8206b925fb0eba3a11839419be118b09105d7b1 > # good: [a8f2800b4f7b76cecb7209cb6a7d2b14904fc711] nfsd4: fix callback > restarting > git bisect good a8f2800b4f7b76cecb7209cb6a7d2b14904fc711 > # bad: [6651149371b842715906311b4631b8489cebf7e8] autofs4: Clean up > autofs4_free_ino() > git bisect bad 6651149371b842715906311b4631b8489cebf7e8 > # good: [0ad53eeefcbb2620b6a71ffdaad4add20b450b8b] afs: add afs_wq and use it > instead of the system workqueue > git bisect good 0ad53eeefcbb2620b6a71ffdaad4add20b450b8b > # good: [01c64feac45cea1317263eabc4f7ee1b240f297f] CIFS: Use d_automount() > rather than abusing follow_link() > git bisect good 01c64feac45cea1317263eabc4f7ee1b240f297f > # good: [b5b801779d59165c4ecf1009009109545bd1f642] autofs4: Add d_manage() > dentry operation > git bisect good b5b801779d59165c4ecf1009009109545bd1f642 > # bad: [e61da20a50d21725ff27571a6dff9468e4fb7146] autofs4: Clean up inode > operations > git bisect bad e61da20a50d21725ff27571a6dff9468e4fb7146 > # good: [8c13a676d5a56495c350f3141824a5ef6c6b4606] autofs4: Remove unused code > git bisect good 8c13a676d5a56495c350f3141824a5ef6c6b4606 I can more or less reproduce this at will now, I think even with very few NFS operations on an automounted nfsv4 mount. Here's an oops from a 2.6.39 kernel: [ 119.419789] tun0: Features changed: 0x00004800 -> 0x00004000 [ 178.242917] FS-Cache: Netfs 'nfs' registered for caching [ 178.269980] SELinux: initialized (dev 0:2c, type nfs4), uses genfs_contexts [ 178.282296] SELinux: initialized (dev 0:2d, type nfs4), uses genfs_contexts [ 523.953284] BUG: Dentry ffff8801f3084180{i=2,n=} still in use (1) [unmount of nfs4 0:2c] [ 523.953306] ------------[ cut here ]------------ [ 523.954013] kernel BUG at fs/dcache.c:925! [ 523.954013] invalid opcode: 0000 [#1] SMP [ 523.954013] last sysfs file: /sys/devices/virtual/bdi/0:45/uevent [ 523.954013] CPU 1 [ 523.954013] Modules linked in: nfs lockd auth_rpcgss nfs_acl tun fuse ip6table_filter ip6_tables ebtable_nat ebtables sunrpc cachefiles fscache cpufreq_ondemand powernow_k8 freq_table mperf it87 adt7475 hwmon_vid xfs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel raid1 snd_hda_codec snd_usb_audio snd_usbmidi_lib snd_hwdep snd_seq snd_rawmidi snd_seq_device snd_pcm snd_timer snd uvcvideo ppdev videodev soundcore media sp5100_tco v4l2_compat_ioctl32 edac_core parport_pc snd_page_alloc i2c_piix4 edac_mce_amd k10temp parport wmi r8169 microcode mii virtio_net kvm_amd kvm ipv6 ata_generic pata_acpi pata_atiixp radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] [ 523.954013] [ 523.954013] Pid: 8387, comm: umount.nfs4 Not tainted 2.6.39-1.fc16.x86_64 #1 BIOSTAR Group TA790GX 128M/TA790GX 128M [ 523.954013] RIP: 0010:[<ffffffff81131788>] [<ffffffff81131788>] shrink_dcache_for_umount_subtree+0x104/0x1c9 [ 523.954013] RSP: 0018:ffff8801f2385d98 EFLAGS: 00010296 [ 523.954013] RAX: 0000000000000062 RBX: ffff8801f3084180 RCX: 00000000000027a7 [ 523.954013] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246 [ 523.954013] RBP: ffff8801f2385dc8 R08: 0000000000000002 R09: 0000ffff00066c0a [ 523.954013] R10: 0000ffff00066c0a R11: 0000000000000003 R12: ffff8801f3084180 [ 523.954013] R13: ffff8801f31305f0 R14: ffff8801f30ec5f0 R15: 0000000000000000 [ 523.954013] FS: 00007f08e468d720(0000) GS:ffff88022fc80000(0000) knlGS:00000000f770f870 [ 523.954013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 523.954013] CR2: 00007fff8fc29ff8 CR3: 00000001f7ac4000 CR4: 00000000000006e0 [ 523.954013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 523.954013] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 523.954013] Process umount.nfs4 (pid: 8387, threadinfo ffff8801f2384000, task ffff8801f7a61710) [ 523.954013] Stack: [ 523.954013] ffff8801fdd74a58 0000000000000000 ffff8801fdd74800 ffff8801f3084240 [ 523.954013] ffff8801f2385f28 ffff8801f30ec540 ffff8801f2385de8 ffffffff81132063 [ 523.954013] ffff8801fdd74800 ffffffffa0542300 ffff8801f2385e18 ffffffff81121f15 [ 523.954013] Call Trace: [ 523.954013] [<ffffffff81132063>] shrink_dcache_for_umount+0x4d/0x5f [ 523.954013] [<ffffffff81121f15>] generic_shutdown_super+0x23/0xde [ 523.954013] [<ffffffff8112204d>] kill_anon_super+0x13/0x4c [ 523.954013] [<ffffffffa051c436>] nfs4_kill_super+0x44/0x77 [nfs] [ 523.954013] [<ffffffff81122271>] deactivate_locked_super+0x26/0x4b [ 523.954013] [<ffffffff81122a83>] deactivate_super+0x37/0x3b [ 523.954013] [<ffffffff811375b3>] mntput_no_expire+0xcc/0xd1 [ 523.954013] [<ffffffff811375de>] mntput+0x26/0x28 [ 523.954013] [<ffffffff81137bef>] release_mounts+0x59/0x73 [ 523.954013] [<ffffffff81138155>] sys_umount+0x28d/0x2da [ 523.954013] [<ffffffff814828c2>] system_call_fastpath+0x16/0x1b [ 523.954013] Code: 8b 40 28 4c 8b 08 48 8b 43 30 48 85 c0 74 07 48 8b 90 a8 00 00 00 48 89 34 24 48 c7 c7 f9 ed 7a 81 48 89 de 31 c0 e8 4e 17 34 00 <0f> 0b 4c 8b 63 18 4c 39 e3 75 11 48 8d bb 90 00 00 00 45 31 e4 [ 523.954013] RIP [<ffffffff81131788>] shrink_dcache_for_umount_subtree+0x104/0x1c9 [ 523.954013] RSP <ffff8801f2385d98> [ 523.954013] [drm] force priority to high [ 523.954013] [drm] force priority to high [ 524.042397] ---[ end trace ff21a3f70c461b5e ]--- [ 534.621786] [drm] force priority to high [ 534.621866] [drm] force priority to high Client rpc stats: calls retrans authrefrsh 44 0 44 Client nfs v4: null read write commit open open_conf 0 0% 0 0% 1 2% 0 0% 1 2% 1 2% open_noat open_dgrd close setattr fsinfo renew 0 0% 0 0% 1 2% 1 2% 4 9% 0 0% setclntid confirm lock lockt locku access 1 2% 1 2% 0 0% 0 0% 0 0% 6 13% getattr lookup lookup_root remove rename link 7 16% 5 11% 1 2% 0 0% 0 0% 0 0% symlink create pathconf statfs readlink readdir 0 0% 0 0% 3 6% 0 0% 0 0% 3 6% server_caps delegreturn getacl setacl fs_locations rel_lkowner 7 16% 0 0% 0 0% 0 0% 0 0% 0 0% exchange_id create_ses destroy_ses sequence get_lease_t reclaim_comp 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% layoutget layoutcommit layoutreturn getdevlist getdevinfo ds_write 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% ds_commit 0 0% That's good because I think it may help narrow down the codepaths that lead to this... All I did was save a file to the mount using claws-mail and wait for autofs to unmount it. I'm going to see if I can narrow down the reproducer somewhat and whether I can take autofs out of the picture. -- Jeff Layton <jlayton@xxxxxxxxxx> -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
