'lo! We saw this in production on one of our shared hosting servers. This is the first time we've seen this particular issue, and it was in the middle of the night when nothing special was happening (that we know of). Linux 2.6.28.7 compiled from source (no patches), NFSv3 in this case, with lockd fairly well-used. There is a large number of NFS mounts to an HA NFS server (same kernel version). After this occurred, our health-checking systems showed that NFS-touching processes seemed to be hanging. Has this issue been seen before? Simon- ...no prior kernel messages for over a day... Apr 13 02:54:38 lsh1001 kernel: ------------[ cut here ]------------ Apr 13 02:54:38 lsh1001 kernel: kernel BUG at kernel/workqueue.c:292! Apr 13 02:54:38 lsh1001 kernel: invalid opcode: 0000 [#1] SMP Apr 13 02:54:38 lsh1001 kernel: last sysfs file: /sys/block/sda/stat Apr 13 02:54:38 lsh1001 kernel: CPU 0 Apr 13 02:54:38 lsh1001 kernel: Modules linked in: nf_conntrack_ftp xt_state xt_owner xt_MARK nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 zlib_inflate Apr 13 02:54:38 lsh1001 kernel: Pid: 1317, comm: rpciod/0 Not tainted 2.6.28.7-hw #1 Apr 13 02:54:38 lsh1001 kernel: RIP: 0010:[<ffffffff80251014>] [<ffffffff80251014>] run_workqueue+0xf4/0x150 Apr 13 02:54:38 lsh1001 kernel: RSP: 0018:ffff88012f3fbe80 EFLAGS: 00010286 Apr 13 02:54:38 lsh1001 kernel: RAX: 0000000000000000 RBX: ffff8801286f3c18 RCX: ffff88012f3fa000 Apr 13 02:54:38 lsh1001 kernel: RDX: ffff8801286f3c18 RSI: ffff88012f3fbec0 RDI: ffff88012f35d200 Apr 13 02:54:38 lsh1001 kernel: RBP: ffff88012f3fbeb0 R08: ffff88012f3fa000 R09: 0000000000000001 Apr 13 02:54:38 lsh1001 kernel: R10: 0000000000000001 R11: 0000000000000001 R12: ffff88012f35d200 Apr 13 02:54:38 lsh1001 kernel: R13: ffff8801286f3c10 R14: ffffffff806706a0 R15: ffff88012f35d208 Apr 13 02:54:38 lsh1001 kernel: FS: 0000000000000000(0000) GS:ffffffff8098a000(0000) knlGS:0000000000000000 Apr 13 02:54:38 lsh1001 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Apr 13 02:54:38 lsh1001 kernel: CR2: 0000000000412de0 CR3: 0000000077cbe000 CR4: 00000000000006e0 Apr 13 02:54:38 lsh1001 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Apr 13 02:54:38 lsh1001 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Apr 13 02:54:38 lsh1001 kernel: Process rpciod/0 (pid: 1317, threadinfo ffff88012f3fa000, task ffff88012f22aae0) Apr 13 02:54:38 lsh1001 kernel: Stack: Apr 13 02:54:38 lsh1001 kernel: ffff88012f3fbeb0 ffff88012f35d218 ffff88012f35d200 ffff88012f3fbec0 Apr 13 02:54:38 lsh1001 kernel: ffff88012f35d208 ffff88012f855ef0 ffff88012f3fbf10 ffffffff80251b73 Apr 13 02:54:38 lsh1001 kernel: 0000000000000000 ffff88012f22aae0 ffffffff80255090 ffff88012f3fbed8 Apr 13 02:54:38 lsh1001 kernel: Call Trace: Apr 13 02:54:38 lsh1001 kernel: [<ffffffff80251b73>] worker_thread+0x93/0xd0 Apr 13 02:54:38 lsh1001 kernel: [<ffffffff80255090>] ? autoremove_wake_function+0x0/0x40 Apr 13 02:54:38 lsh1001 kernel: [<ffffffff80251ae0>] ? worker_thread+0x0/0xd0 Apr 13 02:54:38 lsh1001 kernel: [<ffffffff80254c1d>] kthread+0x4d/0x80 Apr 13 02:54:38 lsh1001 kernel: [<ffffffff8020d659>] child_rip+0xa/0x11 Apr 13 02:54:38 lsh1001 kernel: [<ffffffff80254bd0>] ? kthread+0x0/0x80 Apr 13 02:54:38 lsh1001 kernel: [<ffffffff8020d64f>] ? child_rip+0x0/0x11 Apr 13 02:54:38 lsh1001 kernel: Code: 00 00 00 00 75 81 41 ff 4c 24 48 4c 89 e7 e8 74 5a fd ff 90 fb 0f 1f 80 00 00 00 00 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f c9 c3 <0f> 0b eb fe 65 48 8b 34 25 00 00 00 00 8b 8e 68 01 00 00 48 c7 Apr 13 02:54:38 lsh1001 kernel: RIP [<ffffffff80251014>] run_workqueue+0xf4/0x150 Apr 13 02:54:38 lsh1001 kernel: RSP <ffff88012f3fbe80> Apr 13 02:54:38 lsh1001 kernel: ---[ end trace 22021e591705aaa8 ]--- Apr 13 02:56:38 lsh1001 kernel: lockd: server 10.10.52.220 not responding, still trying Apr 13 02:56:38 lsh1001 kernel: lockd: server 10.10.52.220 not responding, still trying ...no further messages until manual reboot... -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
