xf kernel trace , what is the cause ?

[Gregory Machin <gdm@xxxxxxxxxxxxxx>]

Hi.

 

I have a file server that has had 2 xfs Kernel trace dumps about 2 weeks apart.  

 

The RAID 60 data store is mounted as /home with xfs as the file system. The /home space is shares via NFS export and CIFS share.  At the time of the Trace dumps NFS access hangs and crashed users sessions on the client side.  The output of the last traces is below. 

 

Can you advise as to the cause of this dump and possible resolution ?   

 

System information:

CentOS release 6.5 (Final)

 

Physical

24 GB Ram

Intel(R) Xeon(R) CPU  X5560  @ 2.80GHz

2 x RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)

RAID configured in RAID 6 with software RAID 0 providing final RAID 60

 

Kernel 2.6.32-431.5.1.el6.x86_64
xfsdump-3.0.4-3.el6.x86_64
xfsprogs-3.1.1-14.el6.x86_64

nfs-utils-1.2.3-39.el6.x86_64
nfs4-acl-tools-0.3.3-6.el6.x86_64
nfs-utils-lib-1.1.5-6.el6.x86_64
samba-common-3.6.9-168.el6_5.x86_64
samba4-libs-4.0.0-61.el6_5.rc4.x86_64
samba-3.6.9-168.el6_5.x86_64
samba-winbind-clients-3.6.9-168.el6_5.x86_64
samba-client-3.6.9-168.el6_5.x86_64
samba-winbind-3.6.9-168.el6_5.x86_64

 

[root@nzhmlfpr010 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdc2              97G  3.2G   89G   4% /
tmpfs                  12G     0   12G   0% /dev/shm
/dev/sdc1             504M  100M  379M  21% /boot
/dev/md0               11T  1.7T  9.3T  15% /home

 

[root@nzhmlfpr010 ~]# cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 sdb1[1] sda1[0]
      11707018240 blocks super 1.1 512k chunks

unused devices: <none>

 

Kernel traces:

 

May  2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2529 blocked for more than 120 seconds.
May  2 17:49:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:49:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2529      2 0x00000080
May  2 17:49:39 nzhmlfpr010 kernel: ffff8806384c95e0 0000000000000046 0000000000001000 0000000000000000
May  2 17:49:39 nzhmlfpr010 kernel: 000000000026b000 000000000026c000 0000000000000100 ffffffffffffffff
May  2 17:49:39 nzhmlfpr010 kernel: ffff880623df3ab8 ffff8806384c9fd8 000000000000fbc8 ffff880623df3ab8
May  2 17:49:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2532 blocked for more than 120 seconds.
May  2 17:49:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:49:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2532      2 0x00000080
May  2 17:49:39 nzhmlfpr010 kernel: ffff88063a073430 0000000000000046 0000000000000000 ffff88063a0735d8
May  2 17:49:39 nzhmlfpr010 kernel: 000000043a0735e4 ffff880638681000 ffff88015a9ed838 00ff88063a073580
May  2 17:49:39 nzhmlfpr010 kernel: ffff8806386fbaf8 ffff88063a073fd8 000000000000fbc8 ffff8806386fbaf8
May  2 17:49:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa022faa6>] xfs_read_agf+0x66/0x1a0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa022fc19>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0232642>] xfs_alloc_fix_freelist+0x432/0x4a0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0278187>] ? xfs_trans_add_item+0x57/0x70 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0274938>] ? xfs_perag_get+0x38/0xb0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0232bb6>] xfs_alloc_vextent+0x3d6/0x610 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025e453>] xfs_ialloc_ag_alloc+0x143/0x4e0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025e8cf>] xfs_dialloc+0xdf/0x850 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:49:39 nzhmlfpr010 kernel: INFO: task nfsd:2533 blocked for more than 120 seconds.
May  2 17:49:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:49:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:49:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2533      2 0x00000080
May  2 17:49:39 nzhmlfpr010 kernel: ffff880623f7d8d0 0000000000000046 0000000000000000 ffff880623f7d894
May  2 17:49:39 nzhmlfpr010 kernel: ffff880200000000 ffff88033fc25300 ffff880028276840 0000000000000400
May  2 17:49:39 nzhmlfpr010 kernel: ffff88063875bab8 ffff880623f7dfd8 000000000000fbc8 ffff88063875bab8
May  2 17:49:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0245208>] ? xfs_bmbt_get_all+0x18/0x20 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa023b42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa026342b>] xfs_iunlink+0x4b/0x170 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff810792c7>] ? current_fs_time+0x27/0x30
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b427>] ? xfs_trans_ichgtime+0x27/0xb0 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027b78b>] xfs_droplink+0x5b/0x70 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa027d1ce>] xfs_remove+0x27e/0x390 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811a3767>] ? __d_lookup+0xa7/0x150
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81196203>] ? generic_permission+0x23/0xb0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa028aca8>] xfs_vn_unlink+0x48/0x90 [xfs]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff811979a2>] vfs_unlink+0xa2/0xf0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa055ef5f>] nfsd_unlink+0x1af/0x250 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa056369f>] ? nfsd_cache_lookup+0x59f/0x700 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa05665d3>] nfsd3_proc_remove+0x83/0x120 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:49:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2527 blocked for more than 120 seconds.
May  2 17:51:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:51:39 nzhmlfpr010 kernel: nfsd          D 0000000000000000     0  2527      2 0x00000080
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623ea1600 0000000000000046 0000000000000000 ffffffff8105afee
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623ea1670 ffff880623ea15e0 ffff880623ea1748 0000000200001000
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623e43098 ffff880623ea1fd8 000000000000fbc8 ffff880623e43098
May  2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8105afee>] ? find_busiest_group+0x90e/0x9f0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac27>] xfs_trans_read_buf+0x197/0x410 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d8fe>] xfs_imap_lookup+0x3e/0x170 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8116ecf0>] ? cache_alloc_refill+0x1c0/0x240
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025dbfe>] xfs_imap+0x1ce/0x280 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02650f0>] xfs_iread+0x40/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a59de>] ? inode_init_always+0x11e/0x1c0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025fc8e>] xfs_iget+0x27e/0x6e0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814b121f>] ? tcp_event_new_data_sent+0x4f/0x110
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02860a6>] xfs_nfs_get_inode+0x46/0xe0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028621b>] xfs_fs_fh_to_dentry+0x8b/0x90 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa015271c>] exportfs_decode_fh+0x5c/0x2bc [exportfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055a800>] ? nfsd_acceptable+0x0/0x120 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0560712>] ? exp_find_key+0x62/0xb0 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810ec17e>] ? rcu_start_gp+0x1be/0x230
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055acfa>] fh_verify+0x32a/0x640 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a7c0>] nfsd4_putfh+0x50/0x60 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2528 blocked for more than 120 seconds.
May  2 17:51:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:51:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2528      2 0x00000080
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623ed75e0 0000000000000046 ffff88033745c0f0 0000000019bfcc00
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623ed7690 ffffffffa03d4b5b 0007fffffffff07f ffffffff00000000
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623d6daf8 ffff880623ed7fd8 000000000000fbc8 ffff880623d6daf8
May  2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03d4b5b>] ? tcp_packet+0x68b/0x1750 [nf_conntrack]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81297e28>] ? swiotlb_dma_mapping_error+0x18/0x30
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8116ec8b>] ? cache_alloc_refill+0x15b/0x240
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2529 blocked for more than 120 seconds.
May  2 17:51:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:51:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2529      2 0x00000080
May  2 17:51:39 nzhmlfpr010 kernel: ffff8806384c95e0 0000000000000046 0000000000001000 0000000000000000
May  2 17:51:39 nzhmlfpr010 kernel: 000000000026b000 000000000026c000 0000000000000100 ffffffffffffffff
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623df3ab8 ffff8806384c9fd8 000000000000fbc8 ffff880623df3ab8
May  2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81086036>] ? mod_timer_pending+0x126/0x200
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa03cea95>] ? __nf_ct_refresh_acct+0xe5/0x100 [nf_conntrack]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d503>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025ebc7>] xfs_dialloc+0x3d7/0x850 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2530 blocked for more than 120 seconds.
May  2 17:51:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:51:39 nzhmlfpr010 kernel: nfsd          D 0000000000000000     0  2530      2 0x00000080
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623fbd600 0000000000000046 0000000000000000 ffffffffa0260e6c
May  2 17:51:39 nzhmlfpr010 kernel: 0000000000800000 0002005a00000004 0000000000000003 0000000000000286
May  2 17:51:39 nzhmlfpr010 kernel: ffff880626601058 ffff880623fbdfd8 000000000000fbc8 ffff880626601058
May  2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0260e6c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac27>] xfs_trans_read_buf+0x197/0x410 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d21e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d8fe>] xfs_imap_lookup+0x3e/0x170 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8149a3e8>] ? ip_finish_output+0x148/0x310
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025dbfe>] xfs_imap+0x1ce/0x280 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02650f0>] xfs_iread+0x40/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a59de>] ? inode_init_always+0x11e/0x1c0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025fc8e>] xfs_iget+0x27e/0x6e0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814b121f>] ? tcp_event_new_data_sent+0x4f/0x110
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa02860a6>] xfs_nfs_get_inode+0x46/0xe0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028621b>] xfs_fs_fh_to_dentry+0x8b/0x90 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa015271c>] exportfs_decode_fh+0x5c/0x2bc [exportfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055a800>] ? nfsd_acceptable+0x0/0x120 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0444860>] ? cache_check+0x60/0x360 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0560712>] ? exp_find_key+0x62/0xb0 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81068a53>] ? dequeue_entity+0x113/0x2e0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055acfa>] fh_verify+0x32a/0x640 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa044392d>] ? cache_revisit_request+0x2d/0x120 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a7c0>] nfsd4_putfh+0x50/0x60 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056a458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2531 blocked for more than 120 seconds.
May  2 17:51:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:51:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2531      2 0x00000080
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623131c80 0000000000000046 0000000000000000 00000000329ac000
May  2 17:51:39 nzhmlfpr010 kernel: ffff880338a05200 0000000000000001 ffff8803330c3d88 ffff8803330c3d88
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623d91ab8 ffff880623131fd8 000000000000fbc8 ffff880623d91ab8
May  2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8152938e>] __mutex_lock_slowpath+0x13e/0x180
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8152922b>] mutex_lock+0x2b/0x50
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f102>] nfsd_create_v3+0x102/0x530 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2532 blocked for more than 120 seconds.
May  2 17:51:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:51:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2532      2 0x00000080
May  2 17:51:39 nzhmlfpr010 kernel: ffff88063a073430 0000000000000046 0000000000000000 ffff88063a0735d8
May  2 17:51:39 nzhmlfpr010 kernel: 000000043a0735e4 ffff880638681000 ffff88015a9ed838 00ff88063a073580
May  2 17:51:39 nzhmlfpr010 kernel: ffff8806386fbaf8 ffff88063a073fd8 000000000000fbc8 ffff8806386fbaf8
May  2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa022faa6>] xfs_read_agf+0x66/0x1a0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa022fc19>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0232642>] xfs_alloc_fix_freelist+0x432/0x4a0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0278187>] ? xfs_trans_add_item+0x57/0x70 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0274938>] ? xfs_perag_get+0x38/0xb0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0232bb6>] xfs_alloc_vextent+0x3d6/0x610 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025e453>] xfs_ialloc_ag_alloc+0x143/0x4e0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025e8cf>] xfs_dialloc+0xdf/0x850 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8144b4d4>] ? release_sock+0xe4/0xf0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff814a4aa1>] ? tcp_recvmsg+0x821/0xe80
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0262980>] xfs_ialloc+0x60/0x6e0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028006a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b814>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d8f0>] xfs_create+0x440/0x640 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aa3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028ab80>] xfs_vn_create+0x10/0x20 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811983b6>] vfs_create+0xe6/0x110
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055f3d5>] nfsd_create_v3+0x3d5/0x530 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0566c13>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: INFO: task nfsd:2533 blocked for more than 120 seconds.
May  2 17:51:39 nzhmlfpr010 kernel:      Not tainted 2.6.32-431.5.1.el6.x86_64 #1
May  2 17:51:39 nzhmlfpr010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 17:51:39 nzhmlfpr010 kernel: nfsd          D 0000000000000008     0  2533      2 0x00000080
May  2 17:51:39 nzhmlfpr010 kernel: ffff880623f7d8d0 0000000000000046 0000000000000000 ffff880623f7d894
May  2 17:51:39 nzhmlfpr010 kernel: ffff880200000000 ffff88033fc25300 ffff880028276840 0000000000000400
May  2 17:51:39 nzhmlfpr010 kernel: ffff88063875bab8 ffff880623f7dfd8 000000000000fbc8 ffff88063875bab8
May  2 17:51:39 nzhmlfpr010 kernel: Call Trace:
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81528ac5>] schedule_timeout+0x215/0x2e0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0245208>] ? xfs_bmbt_get_all+0x18/0x20 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa023b42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81529ac2>] __down+0x72/0xb0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] ? _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810a1381>] down+0x41/0x50
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284901>] xfs_buf_lock+0x51/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284a95>] _xfs_buf_find+0xe5/0x230 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0284c14>] xfs_buf_get+0x34/0x1b0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028529c>] xfs_buf_read+0x2c/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027ac87>] xfs_trans_read_buf+0x1f7/0x410 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa025d164>] xfs_read_agi+0x74/0x100 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa026342b>] xfs_iunlink+0x4b/0x170 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff810792c7>] ? current_fs_time+0x27/0x30
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b427>] ? xfs_trans_ichgtime+0x27/0xb0 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027b78b>] xfs_droplink+0x5b/0x70 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa027d1ce>] xfs_remove+0x27e/0x390 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811a3767>] ? __d_lookup+0xa7/0x150
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81196203>] ? generic_permission+0x23/0xb0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa028aca8>] xfs_vn_unlink+0x48/0x90 [xfs]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff811979a2>] vfs_unlink+0xa2/0xf0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa055ef5f>] nfsd_unlink+0x1af/0x250 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa056369f>] ? nfsd_cache_lookup+0x59f/0x700 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa05665d3>] nfsd3_proc_remove+0x83/0x120 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557425>] nfsd_dispatch+0xe5/0x230 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043a844>] svc_process_common+0x344/0x640 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa043ae80>] svc_process+0x110/0x160 [sunrpc]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557b52>] nfsd+0xc2/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffffa0557a90>] ? nfsd+0x0/0x160 [nfsd]
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109aee6>] kthread+0x96/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May  2 17:51:39 nzhmlfpr010 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20

 

 

Thank you

_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs