I have a set of machines which each act solely as an NFS server exporting a single 60Tb XFS filesystem. These machines have suffered from infrequent mysterious crashes since new, but recently a new workload has upped the frequency from monthly to less than a day, enabling me to chase the problem harder. Below is a log, captured by netconsole as the machine was going down hard. I think the crucial line is: [93662.200355] Thread overran stack, or stack corrupted. Based on that I compiled up a kernel patched to set THREAD_ORDER in arch/x86/include/asm/page_64_types.h to 2 (ie 16k kernel stacks.) A machine running that kernel has been sat at load average 130 (128 nfsd threads) for 60 hours now and not seen any problems, without the patch the same load would bring it down in less than a day. I'm fairly convinced that I've fixed my problem, but I guess it's worth posting here as the call trace might enable the devs to find and fix the stack hogs. (And then I won't have to run patched kernels in the future.) More information that might help. x86_64 Storage is fibrechannel attached and the filesystem is hosted on a LVM block device that concatentates four partitions, so the block access is going via a stack of LVM, multipath and Q-logic drivers. Network is Intel 10G ethernet (gxbe driver) Kernel is 2.6.32 with Debian patches. (both kernels) Any other information needed, just let me know. Cheers, Simon. [93662.195788] BUG: scheduling while atomic: nfsd/3686/0xffff8800 [93662.195842] Modules linked in: ioatdma netconsole configfs cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative 8021q garp stp nfsd nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext3 jbd mbcache fuse dm_round_robin dm_multipath scsi_dh autofs4 ohci_hcd sd_mod crc_t10dif usbhid hid snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd ipmi_si ixgbe soundcore psmouse ipmi_msghandler ehci_hcd dca snd_page_alloc uhci_hcd hpilo evdev serio_raw container mdio bnx2 usbcore pcspkr nls_base power_meter qla2xxx scsi_transport_fc scsi_tgt processor button xfs exportfs dm_mirror dm_region_hash dm_log dm_snapshot dm_mod thermal fan thermal_sys cciss scsi_mod [93662.196418] Pid: 3686, comm: nfsd Not tainted 2.6.32-bpo.5-amd64 #1 [93662.196758] Call Trace: [93662.196799] [<ffffffff812fa0a9>] ? schedule+0xce/0x7da [93662.196837] [<ffffffff81176ae4>] ? elv_insert+0xad/0x260 [93662.196871] [<ffffffff812fabeb>] ? schedule_timeout+0x2e/0xdd [93662.196915] [<ffffffffa0058af7>] ? dm_unplug_all+0x3b/0x4c [dm_mod] [93662.196953] [<ffffffff812fb4cf>] ? __down_common+0x8d/0xde [93662.196992] [<ffffffff81068533>] ? down+0x27/0x38 [93662.197053] [<ffffffffa00e2f58>] ? _xfs_buf_find+0x162/0x1e0 [xfs] [93662.197107] [<ffffffffa00e3030>] ? xfs_buf_get_flags+0x5a/0x13b [xfs] [93662.197162] [<ffffffffa00e3123>] ? xfs_buf_read_flags+0x12/0x7a [xfs] [93662.197220] [<ffffffffa00da819>] ? xfs_trans_read_buf+0x189/0x27e [xfs] [93662.197272] [<ffffffffa00a1c09>] ? xfs_read_agf+0x5a/0x149 [xfs] [93662.197322] [<ffffffffa00a1d1a>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs] [93662.197374] [<ffffffffa00a38cf>] ? xfs_alloc_fix_freelist+0x11b/0x3dd [xfs] [93662.197427] [<ffffffffa00a3d57>] ? xfs_alloc_vextent+0x10e/0x3e3 [xfs] [93662.197479] [<ffffffffa00aea39>] ? xfs_bmap_btalloc+0x54f/0x732 [xfs] [93662.197537] [<ffffffffa00b0f4f>] ? xfs_bmapi+0x876/0x104d [xfs] [93662.197594] [<ffffffffa00c6e67>] ? xfs_iext_get_ext+0x34/0x5a [xfs] [93662.197652] [<ffffffffa00cbf3d>] ? xfs_iomap_write_allocate+0x245/0x387 [xfs] [93662.197725] [<ffffffffa00cc9b2>] ? xfs_iomap+0x213/0x285 [xfs] [93662.197763] [<ffffffff8119b5e8>] ? swiotlb_map_sg_attrs+0xeb/0x107 [93662.197817] [<ffffffffa00e012c>] ? xfs_map_blocks+0x25/0x2c [xfs] [93662.197855] [<ffffffff81191960>] ? radix_tree_delete+0xbf/0x1ba [93662.197908] [<ffffffffa00e0d53>] ? xfs_page_state_convert+0x299/0x565 [xfs] [93662.197950] [<ffffffffa005a3bb>] ? dm_table_any_congested+0x66/0xe6 [dm_mod] [93662.198010] [<ffffffffa00e10b7>] ? xfs_vm_releasepage+0x98/0xa5 [xfs] [93662.198065] [<ffffffffa00e129a>] ? xfs_vm_writepage+0xb0/0xe6 [xfs] [93662.198105] [<ffffffff810bdfd5>] ? shrink_page_list+0x375/0x623 [93662.198140] [<ffffffff810be9b8>] ? shrink_list+0x45c/0x767 [93662.198192] [<ffffffffa00b5249>] ? xfs_btree_lookup_get_block+0x9d/0xac [xfs] [93662.198262] [<ffffffffa00b275f>] ? xfs_bmbt_init_key_from_rec+0xc/0x14 [xfs] [93662.198314] [<ffffffffa00b2e1c>] ? xfs_lookup_get_search_key+0x29/0x3c [xfs] [93662.198349] [<ffffffff810bef43>] ? shrink_zone+0x280/0x342 [93662.198381] [<ffffffff810c000a>] ? try_to_free_pages+0x232/0x38e [93662.198413] [<ffffffff810bcfff>] ? isolate_pages_global+0x0/0x20f [93662.198450] [<ffffffff810ba098>] ? __alloc_pages_nodemask+0x3cd/0x5f5 [93662.198486] [<ffffffff810e6535>] ? new_slab+0x42/0x1ca [93662.198516] [<ffffffff810e68ad>] ? __slab_alloc+0x1f0/0x39b [93662.198560] [<ffffffffa00df806>] ? kmem_zone_alloc+0x5e/0xa4 [xfs] [93662.198605] [<ffffffffa00df806>] ? kmem_zone_alloc+0x5e/0xa4 [xfs] [93662.198637] [<ffffffff810e6d88>] ? kmem_cache_alloc+0x7f/0xf0 [93662.198697] [<ffffffffa00df806>] ? kmem_zone_alloc+0x5e/0xa4 [xfs] [93662.198746] [<ffffffffa00df85a>] ? kmem_zone_zalloc+0xe/0x2e [xfs] [93662.198792] [<ffffffffa00d9979>] ? _xfs_trans_alloc+0x29/0x64 [xfs] [93662.198842] [<ffffffffa00d9bb4>] ? xfs_trans_alloc+0x95/0xa1 [xfs] [93662.198888] [<ffffffffa00d9d57>] ? xfs_trans_unlocked_item+0x20/0x3a [xfs] [93662.198931] [<ffffffffa009ab39>] ? xfs_qm_dqattach+0x32/0x3b [xfs] [93662.198978] [<ffffffffa00cbdab>] ? xfs_iomap_write_allocate+0xb3/0x387 [xfs] [93662.199031] [<ffffffffa00cc9b2>] ? xfs_iomap+0x213/0x285 [xfs] [93662.199076] [<ffffffffa00e012c>] ? xfs_map_blocks+0x25/0x2c [xfs] [93662.199122] [<ffffffffa00cca0f>] ? xfs_iomap+0x270/0x285 [xfs] [93662.199169] [<ffffffffa00e0d53>] ? xfs_page_state_convert+0x299/0x565 [xfs] [93662.199218] [<ffffffffa00e129a>] ? xfs_vm_writepage+0xb0/0xe6 [xfs] [93662.199251] [<ffffffff810ba2ca>] ? __writepage+0xa/0x25 [93662.199283] [<ffffffff810ba951>] ? write_cache_pages+0x20b/0x327 [93662.199314] [<ffffffff810ba2c0>] ? __writepage+0x0/0x25 [93662.199347] [<ffffffff810b4925>] ? __filemap_fdatawrite_range+0x4b/0x54 [93662.199381] [<ffffffff810b4954>] ? filemap_write_and_wait_range+0x26/0x52 [93662.199426] [<ffffffffa00e6d82>] ? xfs_write+0x63b/0x6ea [xfs] [93662.199458] [<ffffffff812fb3f5>] ? down_read+0x9/0x19 [93662.199503] [<ffffffffa00c6c36>] ? xfs_iget+0x401/0x45b [xfs] [93662.199550] [<ffffffffa00e36ad>] ? xfs_file_aio_write+0x0/0x5d [xfs] [93662.199584] [<ffffffff810ee511>] ? do_sync_readv_writev+0xc0/0x107 [93662.199630] [<ffffffffa00d9d57>] ? xfs_trans_unlocked_item+0x20/0x3a [xfs] [93662.199666] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e [93662.199698] [<ffffffff810ee39d>] ? rw_copy_check_uvector+0x6d/0xe4 [93662.199733] [<ffffffff810eebad>] ? do_readv_writev+0xb2/0x115 [93662.199788] [<ffffffffa0387f5c>] ? nfsd_setuser_and_check_port+0x62/0x7c [nfsd] [93662.199843] [<ffffffffa038987d>] ? nfsd_vfs_write+0x11a/0x329 [nfsd] [93662.199880] [<ffffffffa038a022>] ? nfsd_open+0x137/0x16c [nfsd] [93662.199915] [<ffffffffa038a34b>] ? nfsd_write+0xc5/0xe2 [nfsd] [93662.199952] [<ffffffffa0390784>] ? nfsd3_proc_write+0xc7/0xe5 [nfsd] [93662.199987] [<ffffffffa0385329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd] [93662.200027] [<ffffffffa02c2513>] ? svc_process+0x403/0x627 [sunrpc] [93662.200067] [<ffffffffa0385772>] ? nfsd+0x0/0x12e [nfsd] [93662.200100] [<ffffffffa0385857>] ? nfsd+0xe5/0x12e [nfsd] [93662.200130] [<ffffffff81064a49>] ? kthread+0x79/0x81 [93662.200162] [<ffffffff81011baa>] ? child_rip+0xa/0x20 [93662.200191] [<ffffffff810649d0>] ? kthread+0x0/0x81 [93662.200220] [<ffffffff81011ba0>] ? child_rip+0x0/0x20 [93662.200256] BUG: unable to handle kernel paging request at 000000006eab11a0 [93662.200294] IP: [<ffffffff8103fd9b>] update_curr+0xf9/0x147 [93662.200331] PGD 0 [93662.200355] Thread overran stack, or stack corrupted [93662.200383] Oops: 0000 [#1] SMP [93662.200414] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/0000:07:00.1/irq [93662.200461] CPU 1 [93662.200486] Modules linked in: ioatdma netconsole configfs cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative 8021q garp stp nfsd nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext3 jbd mbcache fuse dm_round_robin dm_multipath scsi_dh autofs4 ohci_hcd sd_mod crc_t10dif usbhid hid snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd ipmi_si ixgbe soundcore psmouse ipmi_msghandler ehci_hcd dca snd_page_alloc uhci_hcd hpilo evdev serio_raw container mdio bnx2 usbcore pcspkr nls_base power_meter qla2xxx scsi_transport_fc scsi_tgt processor button xfs exportfs dm_mirror dm_region_hash dm_log dm_snapshot dm_mod [93662.201420] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [93662.201888] [<ffffffff812fb4cf>] ? __down_common+0x8d/0xde [93662.208496] [<ffffffff81068533>] ? down+0x27/0x38 [93662.208539] [<ffffffffa00e2f58>] ? _xfs_buf_find+0x162/0x1e0 [xfs] [93662.208584] [<ffffffffa00e3030>] ? xfs_buf_get_flags+0x5a/0x13b [xfs] [93662.208629] [<ffffffffa00e3123>] ? xfs_buf_read_flags+0x12/0x7a [xfs] [93662.208675] [<ffffffffa00da819>] ? xfs_trans_read_buf+0x189/0x27e [xfs] [93662.208720] [<ffffffffa00a1c09>] ? xfs_read_agf+0x5a/0x149 [xfs] [93662.208762] [<ffffffffa00a1d1a>] ? xfs_alloc_read_agf+0x22/0xa4 [xfs] [93662.208806] [<ffffffffa00a38cf>] ? xfs_alloc_fix_freelist+0x11b/0x3dd [xfs] [93662.208851] [<ffffffffa00a3d57>] ? xfs_alloc_vextent+0x10e/0x3e3 [xfs] [93662.208896] [<ffffffffa00aea39>] ? xfs_bmap_btalloc+0x54f/0x732 [xfs] [93662.208945] [<ffffffffa00b0f4f>] ? xfs_bmapi+0x876/0x104d [xfs] [93662.208995] [<ffffffffa00c6e67>] ? xfs_iext_get_ext+0x34/0x5a [xfs] [93662.209042] [<ffffffffa00cbf3d>] ? xfs_iomap_write_allocate+0x245/0x387 [xfs] [93662.209108] [<ffffffffa00cc9b2>] ? xfs_iomap+0x213/0x285 [xfs] [93662.209424] [<ffffffff810bdfd5>] ? shrink_page_list+0x375/0x623 [93662.209757] [<ffffffff810ba098>] ? __alloc_pages_nodemask+0x3cd/0x5f5 [93662.210078] [<ffffffffa00d9979>] ? _xfs_trans_alloc+0x29/0x64 [xfs] [93662.210575] [<ffffffff810ba951>] ? write_cache_pages+0x20b/0x327 [93662.211100] [<ffffffffa038987d>] ? nfsd_vfs_write+0x11a/0x329 [nfsd] 48 [93662.211857] CR2: 000000006eab11a0 [93662.212189] Kernel panic - not syncing: Fatal exception in interrupt [93662.212497] [<ffffffff8104e387>] ? release_console_sem+0x17e/0x1af [93662.212818] [<ffffffff81011ba0>] ? child_rip+0x0/0x20 [93662.213328] [<ffffffffa0058af7>] ? dm_unplug_all+0x3b/0x4c [dm_mod] [93662.213841] [<ffffffffa00a1c09>] ? xfs_read_agf+0x5a/0x149 [xfs] [93662.214384] [<ffffffffa00cbf3d>] ? xfs_iomap_write_allocate+0x245/0x387 [xfs] [93662.214831] [<ffffffffa005a3bb>] ? dm_table_any_congested+0x66/0xe6 [dm_mod] [93662.215447] [<ffffffff810bef43>] ? shrink_zone+0x280/0x342 [93662.215703] [<ffffffff810e6535>] ? new_slab+0x42/0x1ca [93662.216138] [<ffffffffa00df85a>] ? kmem_zone_zalloc+0xe/0x2e [xfs] [93662.216448] [<ffffffffa009ab39>] ? xfs_qm_dqattach+0x32/0x3b [xfs] [93662.217403] [<ffffffff812fb3f5>] ? down_read+0x9/0x19 [93662.217698] [<ffffffffa00d9d57>] ? xfs_trans_unlocked_item+0x20/0x3a [xfs] [93662.218208] [<ffffffffa038a34b>] ? nfsd_write+0xc5/0xe2 [nfsd] [93662.218482] [<ffffffffa0385772>] ? nfsd+0x0/0x12e [nfsd] [93662.218732] [<ffffffff810649d0>] ? kthread+0x0/0x81 [93662.225424] [<ffffffff81011ba0>] ? child_rip+0x0/0x20 _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
