On Wed, Sep 24, 2014 at 4:52 PM, Micha Krause <micha at krausam.de> wrote:
> Hi,
>
>> Like I mentioned in my other reply, I'd be very interested in any
>>
>> similar messages on kernel other than 3.15.*, 3.16.1 and 3.16.2. One
>> hung task stack trace is usually not enough to diagnose this sort of
>> problems.
>
>
> Ok, here is a more complete dmesg output from kernel 3.14:
>
> [ 22.250600] rbd: loaded
> [ 23.159914] libceph: client24407525 fsid
> 46e857ee-855c-4165-8413-8950f8d081be
> [ 23.289691] libceph: mon1 10.210.34.11:6789 session established
> [ 23.890625] rbd2: unknown partition table
> [ 23.890702] rbd: rbd2: added with size 0x10000000000
> [ 23.937051] rbd0: unknown partition table
> [ 23.937144] rbd: rbd0: added with size 0x10000000000
> [ 24.052402] rbd1: unknown partition table
> [ 24.052479] rbd: rbd1: added with size 0xa0000000000
> [ 24.396333] rbd3: unknown partition table
> [ 24.396430] rbd: rbd3: added with size 0x10000000000
> [ 25.927373] SGI XFS with ACLs, security attributes, realtime, large
> block/inode numbers, no debug enabled
> [ 25.960975] XFS (rbd1): Mounting Filesystem
> [ 25.961072] XFS (rbd3): Mounting Filesystem
> [ 25.961637] XFS (rbd2): Mounting Filesystem
> [ 25.961708] XFS (rbd0): Mounting Filesystem
> [ 28.236952] XFS (rbd3): Starting recovery (logdev: internal)
> [ 28.794631] XFS (rbd1): Starting recovery (logdev: internal)
> [ 31.501516] XFS (rbd0): Starting recovery (logdev: internal)
> [ 35.498950] XFS (rbd2): Starting recovery (logdev: internal)
> [ 63.601465] XFS (rbd0): Ending recovery (logdev: internal)
> [ 64.214852] XFS (rbd3): Ending recovery (logdev: internal)
> [ 64.783531] rbd4: unknown partition table
> [ 64.784005] rbd: rbd4: added with size 0x10000000000
> [ 65.280960] XFS (rbd4): Mounting Filesystem
> [ 68.443439] XFS (rbd2): Ending recovery (logdev: internal)
> [ 69.030358] XFS (rbd4): Starting recovery (logdev: internal)
> [ 69.945523] rbd5: unknown partition table
> [ 69.946021] rbd: rbd5: added with size 0x10000000000
> [ 70.398567] XFS (rbd5): Mounting Filesystem
> [ 71.187934] XFS (rbd5): Starting recovery (logdev: internal)
> [ 74.144173] rbd6: unknown partition table
> [ 74.144630] rbd: rbd6: added with size 0x10000000000
> [ 75.402767] XFS (rbd6): Mounting Filesystem
> [ 76.133654] XFS (rbd6): Starting recovery (logdev: internal)
> [ 111.131893] XFS (rbd4): Ending recovery (logdev: internal)
> [ 112.460383] rbd7: unknown partition table
> [ 112.460898] rbd: rbd7: added with size 0x10000000000
> [ 116.834457] XFS (rbd5): Ending recovery (logdev: internal)
> [ 116.949218] XFS (rbd6): Ending recovery (logdev: internal)
> [ 166.357039] XFS (rbd1): Ending recovery (logdev: internal)
> [ 167.531353] XFS (rbd7): Mounting Filesystem
> [ 168.303166] XFS (rbd7): Starting recovery (logdev: internal)
> [ 172.477811] XFS (rbd7): Ending recovery (logdev: internal)
> [ 2038.723394] INFO: task kthreadd:2 blocked for more than 120 seconds.
> [ 2038.723497] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.723553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.723637] kthreadd D ffff88003fc14340 0 2 0
> 0x00000000
> [ 2038.723641] ffff88003fa3a8e0 0000000000000046 ffff88003fa43628
> ffffffff81813480
> [ 2038.723644] 0000000000014340 ffff88003fa43fd8 0000000000014340
> ffff88003fa3a8e0
> [ 2038.723646] ffff88003fa43638 ffff8800006a7410 7fffffffffffffff
> ffff8800006a7408
> [ 2038.723648] Call Trace:
> [ 2038.723660] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.723665] [<ffffffff8127300b>] ? blk_finish_plug+0xb/0x30
> [ 2038.723677] [<ffffffffa048c957>] ? _xfs_buf_ioapply+0x277/0x2e0 [xfs]
> [ 2038.723680] [<ffffffff814f0b94>] ? wait_for_completion+0xa4/0x110
> [ 2038.723685] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.723691] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs]
> [ 2038.723696] [<ffffffffa048cf56>] ? xfs_buf_iowait+0x96/0xf0 [xfs]
> [ 2038.723703] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs]
> [ 2038.723711] [<ffffffffa0494b24>] ? xfs_reclaim_inode+0x2f4/0x310 [xfs]
> [ 2038.723720] [<ffffffffa0494d37>] ? xfs_reclaim_inodes_ag+0x1f7/0x320
> [xfs]
> [ 2038.723729] [<ffffffffa04959dc>] ? xfs_reclaim_inodes_nr+0x2c/0x40 [xfs]
> [ 2038.723736] [<ffffffff8119b9f7>] ? super_cache_scan+0x167/0x170
> [ 2038.723742] [<ffffffff8113e406>] ? shrink_slab_node+0x126/0x290
> [ 2038.723746] [<ffffffff811923b3>] ? vmpressure+0x23/0xa0
> [ 2038.723750] [<ffffffff81140862>] ? shrink_slab+0x82/0x130
> [ 2038.723755] [<ffffffff81143722>] ? do_try_to_free_pages+0x3c2/0x510
> [ 2038.723762] [<ffffffff81136cff>] ? get_page_from_freelist+0x59f/0x8a0
> [ 2038.723766] [<ffffffff81143bd7>] ? try_to_free_pages+0x107/0x1f0
> [ 2038.723770] [<ffffffff811376f0>] ? __alloc_pages_nodemask+0x6f0/0xaf0
> [ 2038.723776] [<ffffffff810624eb>] ? copy_process+0x20b/0x1ba0
> [ 2038.723782] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.723788] [<ffffffff8109dea1>] ? dequeue_task_fair+0x231/0x7f0
> [ 2038.723795] [<ffffffff810145af>] ? __switch_to+0x12f/0x4e0
> [ 2038.723798] [<ffffffff8109a745>] ? set_next_entity+0x35/0x80
> [ 2038.723802] [<ffffffff81063fad>] ? do_fork+0x6d/0x340
> [ 2038.723805] [<ffffffff810642a1>] ? kernel_thread+0x21/0x30
> [ 2038.723809] [<ffffffff810872e5>] ? kthreadd+0x135/0x190
> [ 2038.723813] [<ffffffff810871b0>] ? kthread_create_on_cpu+0x60/0x60
> [ 2038.723817] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.723821] [<ffffffff810871b0>] ? kthread_create_on_cpu+0x60/0x60
> [ 2038.723828] INFO: task kswapd0:23 blocked for more than 120 seconds.
> [ 2038.723918] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.724002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.724111] kswapd0 D ffff88003fc14340 0 23 2
> 0x00000000
> [ 2038.724115] ffff88003bb8aaa0 0000000000000046 0000003300000000
> ffff88002f2502d0
> [ 2038.724117] 0000000000014340 ffff88003bb91fd8 0000000000014340
> ffff88003bb8aaa0
> [ 2038.724118] 0000000000000000 ffff88003bb91610 7fffffffffffffff
> ffff88003bb91608
> [ 2038.724120] Call Trace:
> [ 2038.724125] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.724134] [<ffffffffa04dc6b8>] ? xfs_iext_bno_to_ext+0x88/0x160 [xfs]
> [ 2038.724137] [<ffffffff810968fb>] ? try_to_wake_up+0xcb/0x280
> [ 2038.724140] [<ffffffff814f0b94>] ? wait_for_completion+0xa4/0x110
> [ 2038.724143] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.724148] [<ffffffffa0488eed>] ? xfs_bmapi_allocate+0x8d/0xb0 [xfs]
> [ 2038.724156] [<ffffffffa04bafa8>] ? xfs_bmapi_write+0x458/0x770 [xfs]
> [ 2038.724160] [<ffffffff8107e86b>] ? __queue_work+0x14b/0x3b0
> [ 2038.724165] [<ffffffffa04883f0>] ? xfs_bmap_count_tree+0x1f0/0x1f0 [xfs]
> [ 2038.724171] [<ffffffffa0499b38>] ? xfs_iomap_write_allocate+0x148/0x360
> [xfs]
> [ 2038.724176] [<ffffffffa0485025>] ? xfs_map_blocks+0x195/0x270 [xfs]
> [ 2038.724181] [<ffffffffa04862df>] ? xfs_vm_writepage+0x20f/0x570 [xfs]
> [ 2038.724184] [<ffffffff811647d0>] ? __page_check_address+0x1f0/0x1f0
> [ 2038.724187] [<ffffffff811418a0>] ? shrink_page_list+0x790/0xa60
> [ 2038.724194] [<ffffffffa04e5753>] ? xfs_perag_get_tag+0x43/0x100 [xfs]
> [ 2038.724197] [<ffffffff811421dd>] ? shrink_inactive_list+0x19d/0x510
> [ 2038.724200] [<ffffffff81142c00>] ? shrink_lruvec+0x330/0x630
> [ 2038.724203] [<ffffffff81142f6e>] ? shrink_zone+0x6e/0x1a0
> [ 2038.724205] [<ffffffff811441ab>] ? balance_pgdat+0x38b/0x5c0
> [ 2038.724208] [<ffffffff81070397>] ? try_to_del_timer_sync+0x47/0x60
> [ 2038.724211] [<ffffffff81144547>] ? kswapd+0x167/0x460
> [ 2038.724214] [<ffffffff810a67d0>] ? __wake_up_sync+0x10/0x10
> [ 2038.724217] [<ffffffff811443e0>] ? balance_pgdat+0x5c0/0x5c0
> [ 2038.724219] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.724221] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.724223] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.724226] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.724244] INFO: task nfsd:2740 blocked for more than 120 seconds.
> [ 2038.724317] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.724373] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.724456] nfsd D ffff88003fc14340 0 2740 2
> 0x00000000
> [ 2038.724459] ffff88003b794350 0000000000000046 ffff88003b9a5668
> ffff88002f2502d0
> [ 2038.724461] 0000000000014340 ffff88003b9a5fd8 0000000000014340
> ffff88003b794350
> [ 2038.724463] ffff88003b9a5678 ffff880009997610 7fffffffffffffff
> ffff880009997608
> [ 2038.724465] Call Trace:
> [ 2038.724468] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.724471] [<ffffffff8127300b>] ? blk_finish_plug+0xb/0x30
> [ 2038.724476] [<ffffffffa048c957>] ? _xfs_buf_ioapply+0x277/0x2e0 [xfs]
> [ 2038.724479] [<ffffffff814f0b94>] ? wait_for_completion+0xa4/0x110
> [ 2038.724482] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.724487] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs]
> [ 2038.724492] [<ffffffffa048cf56>] ? xfs_buf_iowait+0x96/0xf0 [xfs]
> [ 2038.724498] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs]
> [ 2038.724503] [<ffffffffa0494b24>] ? xfs_reclaim_inode+0x2f4/0x310 [xfs]
> [ 2038.724509] [<ffffffffa0494d37>] ? xfs_reclaim_inodes_ag+0x1f7/0x320
> [xfs]
> [ 2038.724515] [<ffffffffa04959dc>] ? xfs_reclaim_inodes_nr+0x2c/0x40 [xfs]
> [ 2038.724518] [<ffffffff8119b9f7>] ? super_cache_scan+0x167/0x170
> [ 2038.724520] [<ffffffff8113e406>] ? shrink_slab_node+0x126/0x290
> [ 2038.724523] [<ffffffff811923b3>] ? vmpressure+0x23/0xa0
> [ 2038.724525] [<ffffffff81140862>] ? shrink_slab+0x82/0x130
> [ 2038.724527] [<ffffffff81143722>] ? do_try_to_free_pages+0x3c2/0x510
> [ 2038.724530] [<ffffffff81143bd7>] ? try_to_free_pages+0x107/0x1f0
> [ 2038.724533] [<ffffffff811376f0>] ? __alloc_pages_nodemask+0x6f0/0xaf0
> [ 2038.724537] [<ffffffff81176365>] ? alloc_pages_current+0xb5/0x180
> [ 2038.724548] [<ffffffffa035a0bc>] ? svc_recv+0xbc/0xa10 [sunrpc]
> [ 2038.724552] [<ffffffffa0358f65>] ? svc_xprt_put+0x5/0x20 [sunrpc]
> [ 2038.724557] [<ffffffffa035aae0>] ? svc_send+0xd0/0x100 [sunrpc]
> [ 2038.724563] [<ffffffffa03e66c5>] ? nfsd+0xa5/0x130 [nfsd]
> [ 2038.724566] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.724569] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.724571] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.724573] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.724575] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.724577] INFO: task nfsd:2741 blocked for more than 120 seconds.
> [ 2038.724641] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.724696] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.724819] nfsd D ffff88003fc14340 0 2741 2
> 0x00000000
> [ 2038.724823] ffff88003b794c20 0000000000000046 0000000000000008
> ffff8800176f4b60
> [ 2038.724826] 0000000000014340 ffff88003b9e7fd8 0000000000014340
> ffff88003b794c20
> [ 2038.724828] 00000000000006f4 ffff88003b794c20 fffffffe00000001
> ffff88000a3df890
> [ 2038.724831] Call Trace:
> [ 2038.724835] [<ffffffff814f2145>] ? rwsem_down_write_failed+0x105/0x1c0
> [ 2038.724839] [<ffffffff812a53e3>] ?
> call_rwsem_down_write_failed+0x13/0x20
> [ 2038.724841] [<ffffffff814f1a34>] ? down_write+0x24/0x30
> [ 2038.724848] [<ffffffffa049b012>] ? xfs_setattr_nonsize+0x1c2/0x5c0 [xfs]
> [ 2038.724855] [<ffffffffa049b7eb>] ? xfs_vn_setattr+0x2b/0x70 [xfs]
> [ 2038.724858] [<ffffffff811b309b>] ? notify_change+0x1eb/0x3d0
> [ 2038.724863] [<ffffffffa03ed5fc>] ? nfsd_setattr+0x24c/0x4b0 [nfsd]
> [ 2038.724867] [<ffffffffa03f5653>] ? nfsd3_proc_setattr+0x73/0xc0 [nfsd]
> [ 2038.724871] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
> [ 2038.724876] [<ffffffffa034ad64>] ? svc_process_common+0x354/0x690
> [sunrpc]
> [ 2038.724879] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.724883] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 [sunrpc]
> [ 2038.724886] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd]
> [ 2038.724890] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.724892] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.724894] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.724896] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.724898] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.724900] INFO: task nfsd:2742 blocked for more than 120 seconds.
> [ 2038.724977] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.725221] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.725314] nfsd D ffff88003fc14340 0 2742 2
> 0x00000000
> [ 2038.725316] ffff88003b7162d0 0000000000000046 ffff880025460060
> ffff88002f2502d0
> [ 2038.725318] 0000000000014340 ffff88003c709fd8 0000000000014340
> ffff88003b7162d0
> [ 2038.725320] ffff88003c709810 ffff88000dab4370 7fffffffffffffff
> 7fffffffffffffff
> [ 2038.725322] Call Trace:
> [ 2038.725326] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.725333] [<ffffffffa04b150a>] ? xfs_bmap_search_extents+0x6a/0xf0
> [xfs]
> [ 2038.725336] [<ffffffff814f193c>] ? __down_common+0x97/0xea
> [ 2038.725341] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.725344] [<ffffffff810aa697>] ? down+0x37/0x40
> [ 2038.725349] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs]
> [ 2038.725354] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.725360] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs]
> [ 2038.725365] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 [xfs]
> [ 2038.725368] [<ffffffff814f2719>] ? _raw_spin_unlock_irqrestore+0x9/0x10
> [ 2038.725375] [<ffffffffa04f1112>] ? xfs_trans_read_buf_map+0x282/0x4f0
> [xfs]
> [ 2038.725382] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 [xfs]
> [ 2038.725386] [<ffffffff81401e9e>] ? dev_hard_start_xmit+0x33e/0x5f0
> [ 2038.725393] [<ffffffffa04c95ef>] ? xfs_dir3_block_read+0x2f/0x70 [xfs]
> [ 2038.725400] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs]
> [ 2038.725406] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs]
> [ 2038.725413] [<ffffffffa04c9668>] ? xfs_dir2_block_lookup_int+0x38/0x1f0
> [xfs]
> [ 2038.725419] [<ffffffffa049530e>] ? xfs_iget+0x4ae/0x810 [xfs]
> [ 2038.725425] [<ffffffffa04ca094>] ? xfs_dir2_block_lookup+0x24/0x120
> [xfs]
> [ 2038.725432] [<ffffffffa04c86c0>] ? xfs_dir2_isblock+0x20/0x60 [xfs]
> [ 2038.725438] [<ffffffffa04c8cd0>] ? xfs_dir_lookup+0xf0/0x1a0 [xfs]
> [ 2038.725445] [<ffffffffa04d8790>] ? xfs_lookup+0xe0/0x140 [xfs]
> [ 2038.725448] [<ffffffff811af96a>] ? __d_alloc+0x13a/0x180
> [ 2038.725454] [<ffffffffa049a120>] ? xfs_vn_lookup+0x50/0x90 [xfs]
> [ 2038.725457] [<ffffffff811a2299>] ? generic_permission+0xf9/0x1a0
> [ 2038.725459] [<ffffffff811a1c64>] ? lookup_real+0x14/0x50
> [ 2038.725462] [<ffffffff811a2142>] ? __lookup_hash+0x32/0x50
> [ 2038.725464] [<ffffffff811a65dd>] ? lookup_one_len+0xbd/0x110
> [ 2038.725469] [<ffffffffa03eb3e7>] ? nfsd_lookup_dentry+0x127/0x490 [nfsd]
> [ 2038.725474] [<ffffffffa03eb7cb>] ? nfsd_lookup+0x7b/0x160 [nfsd]
> [ 2038.725481] [<ffffffffa034e3e0>] ? svcauth_null_release+0x60/0x60
> [sunrpc]
> [ 2038.725487] [<ffffffffa03f5531>] ? nfsd3_proc_lookup+0xd1/0x180 [nfsd]
> [ 2038.725491] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
> [ 2038.725498] [<ffffffffa034ad64>] ? svc_process_common+0x354/0x690
> [sunrpc]
> [ 2038.725502] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.725507] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 [sunrpc]
> [ 2038.725511] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd]
> [ 2038.725514] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.725517] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.725519] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.725521] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.725524] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.725526] INFO: task nfsd:2743 blocked for more than 120 seconds.
> [ 2038.725589] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.725643] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.725735] nfsd D ffff88003fc14340 0 2743 2
> 0x00000000
> [ 2038.725739] ffff88003d468a60 0000000000000046 0000000000002000
> ffffffff81813480
> [ 2038.725742] 0000000000014340 ffff88003c72dfd8 0000000000014340
> ffff88003d468a60
> [ 2038.725744] ffff880037acd1c0 ffff880039d78870 7fffffffffffffff
> 7fffffffffffffff
> [ 2038.725747] Call Trace:
> [ 2038.725752] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.725755] [<ffffffff814f193c>] ? __down_common+0x97/0xea
> [ 2038.725763] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.725768] [<ffffffff810aa697>] ? down+0x37/0x40
> [ 2038.725775] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs]
> [ 2038.725782] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.725790] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs]
> [ 2038.725797] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 [xfs]
> [ 2038.725801] [<ffffffff814f2719>] ? _raw_spin_unlock_irqrestore+0x9/0x10
> [ 2038.725811] [<ffffffffa04f1112>] ? xfs_trans_read_buf_map+0x282/0x4f0
> [xfs]
> [ 2038.725821] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 [xfs]
> [ 2038.725824] [<ffffffff81070108>] ? internal_add_timer+0x18/0x50
> [ 2038.725831] [<ffffffffa04c95ef>] ? xfs_dir3_block_read+0x2f/0x70 [xfs]
> [ 2038.725838] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs]
> [ 2038.725843] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs]
> [ 2038.725850] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs]
> [ 2038.725855] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs]
> [ 2038.725861] [<ffffffffa04c9668>] ? xfs_dir2_block_lookup_int+0x38/0x1f0
> [xfs]
> [ 2038.725867] [<ffffffffa049530e>] ? xfs_iget+0x4ae/0x810 [xfs]
> [ 2038.725873] [<ffffffffa04ca094>] ? xfs_dir2_block_lookup+0x24/0x120
> [xfs]
> [ 2038.725879] [<ffffffffa04c86c0>] ? xfs_dir2_isblock+0x20/0x60 [xfs]
> [ 2038.725885] [<ffffffffa04c8cd0>] ? xfs_dir_lookup+0xf0/0x1a0 [xfs]
> [ 2038.725892] [<ffffffffa04d8790>] ? xfs_lookup+0xe0/0x140 [xfs]
> [ 2038.725894] [<ffffffff811af96a>] ? __d_alloc+0x13a/0x180
> [ 2038.725901] [<ffffffffa049a120>] ? xfs_vn_lookup+0x50/0x90 [xfs]
> [ 2038.725903] [<ffffffff811a1c64>] ? lookup_real+0x14/0x50
> [ 2038.725905] [<ffffffff811a2142>] ? __lookup_hash+0x32/0x50
> [ 2038.725908] [<ffffffff811a65dd>] ? lookup_one_len+0xbd/0x110
> [ 2038.725912] [<ffffffffa03eb3e7>] ? nfsd_lookup_dentry+0x127/0x490 [nfsd]
> [ 2038.725916] [<ffffffffa03eb7cb>] ? nfsd_lookup+0x7b/0x160 [nfsd]
> [ 2038.725920] [<ffffffffa034e3e0>] ? svcauth_null_release+0x60/0x60
> [sunrpc]
> [ 2038.725924] [<ffffffffa03f5531>] ? nfsd3_proc_lookup+0xd1/0x180 [nfsd]
> [ 2038.725928] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
> [ 2038.725932] [<ffffffffa034ad64>] ? svc_process_common+0x354/0x690
> [sunrpc]
> [ 2038.725935] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.725939] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 [sunrpc]
> [ 2038.725944] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd]
> [ 2038.725949] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.725952] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.725956] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.725959] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.725963] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.725966] INFO: task nfsd:2744 blocked for more than 120 seconds.
> [ 2038.726054] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.726125] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.726244] nfsd D ffff88003fc14340 0 2744 2
> 0x00000000
> [ 2038.726248] ffff88003d1d0d20 0000000000000046 ffff88001f3849a8
> ffff88002f2502d0
> [ 2038.726251] 0000000000014340 ffff88003c74ffd8 0000000000014340
> ffff88003d1d0d20
> [ 2038.726253] 0000000000000001 ffff88001be09970 7fffffffffffffff
> 7fffffffffffffff
> [ 2038.726256] Call Trace:
> [ 2038.726261] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.726266] [<ffffffff814f193c>] ? __down_common+0x97/0xea
> [ 2038.726274] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.726278] [<ffffffff810aa697>] ? down+0x37/0x40
> [ 2038.726285] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs]
> [ 2038.726293] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.726300] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs]
> [ 2038.726308] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 [xfs]
> [ 2038.726318] [<ffffffffa04f1112>] ? xfs_trans_read_buf_map+0x282/0x4f0
> [xfs]
> [ 2038.726327] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 [xfs]
> [ 2038.726337] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs]
> [ 2038.726345] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs]
> [ 2038.726354] [<ffffffffa04cc4ee>] ?
> xfs_dir3_leaf_read.constprop.13+0x2e/0x80 [xfs]
> [ 2038.726357] [<ffffffff8114d31d>] ? zone_statistics+0x9d/0xa0
> [ 2038.726366] [<ffffffffa04cd5df>] ? xfs_dir2_leaf_lookup_int+0x4f/0x2d0
> [xfs]
> [ 2038.726373] [<ffffffffa049530e>] ? xfs_iget+0x4ae/0x810 [xfs]
> [ 2038.726382] [<ffffffffa04cde39>] ? xfs_dir2_leaf_lookup+0x29/0x140 [xfs]
> [ 2038.726390] [<ffffffffa04c8720>] ? xfs_dir2_isleaf+0x20/0x60 [xfs]
> [ 2038.726396] [<ffffffffa03e9a20>] ? _fh_update.isra.9.part.10+0x50/0x50
> [nfsd]
> [ 2038.726404] [<ffffffffa04c8d6a>] ? xfs_dir_lookup+0x18a/0x1a0 [xfs]
> [ 2038.726413] [<ffffffffa04d8790>] ? xfs_lookup+0xe0/0x140 [xfs]
> [ 2038.726417] [<ffffffff811af96a>] ? __d_alloc+0x13a/0x180
> [ 2038.726425] [<ffffffffa049a120>] ? xfs_vn_lookup+0x50/0x90 [xfs]
> [ 2038.726428] [<ffffffff811a1c64>] ? lookup_real+0x14/0x50
> [ 2038.726432] [<ffffffff811a2142>] ? __lookup_hash+0x32/0x50
> [ 2038.726435] [<ffffffff811a65dd>] ? lookup_one_len+0xbd/0x110
> [ 2038.726441] [<ffffffffa03eda76>] ? do_nfsd_create+0x216/0x610 [nfsd]
> [ 2038.726446] [<ffffffffa03f4f4d>] ? nfsd3_proc_create+0x16d/0x250 [nfsd]
> [ 2038.726450] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
> [ 2038.726457] [<ffffffffa034ad64>] ? svc_process_common+0x354/0x690
> [sunrpc]
> [ 2038.726461] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.726466] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 [sunrpc]
> [ 2038.726470] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd]
> [ 2038.726475] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.726478] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.726481] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.726485] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.726488] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.726491] INFO: task nfsd:2745 blocked for more than 120 seconds.
> [ 2038.726579] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.726657] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.726773] nfsd D ffff88003fc14340 0 2745 2
> 0x00000000
> [ 2038.726777] ffff88003d1e0da0 0000000000000046 ffff880005900800
> ffff88002f2502d0
> [ 2038.726780] 0000000000014340 ffff88003c773fd8 0000000000014340
> ffff88003d1e0da0
> [ 2038.726782] 0000000100000003 ffff8800063930f0 7fffffffffffffff
> 7fffffffffffffff
> [ 2038.726785] Call Trace:
> [ 2038.726790] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.726794] [<ffffffff814f193c>] ? __down_common+0x97/0xea
> [ 2038.726802] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.726805] [<ffffffff810aa697>] ? down+0x37/0x40
> [ 2038.726812] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs]
> [ 2038.726820] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.726826] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs]
> [ 2038.726834] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 [xfs]
> [ 2038.726844] [<ffffffffa04f11da>] ? xfs_trans_read_buf_map+0x34a/0x4f0
> [xfs]
> [ 2038.726853] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 [xfs]
> [ 2038.726862] [<ffffffffa04c57a9>] ? xfs_da3_node_read+0x19/0xc0 [xfs]
> [ 2038.726868] [<ffffffff8142163c>] ? sch_direct_xmit+0x7c/0x1d0
> [ 2038.726878] [<ffffffffa04c6d1e>] ? xfs_da3_node_lookup_int+0x5e/0x2f0
> [xfs]
> [ 2038.726887] [<ffffffffa04d1777>] ? xfs_dir2_node_removename+0x77/0x6d0
> [xfs]
> [ 2038.726894] [<ffffffffa0354f2c>] ? cache_check+0x5c/0x360 [sunrpc]
> [ 2038.726903] [<ffffffffa04b8fee>] ? xfs_bmap_last_extent+0x6e/0x90 [xfs]
> [ 2038.726912] [<ffffffffa04b9127>] ? xfs_bmap_last_offset+0x87/0xc0 [xfs]
> [ 2038.726920] [<ffffffffa04c8bd5>] ? xfs_dir_removename+0x185/0x190 [xfs]
> [ 2038.726930] [<ffffffffa04dadf3>] ? xfs_remove+0x2c3/0x410 [xfs]
> [ 2038.726936] [<ffffffff8106d5e3>] ? ns_capable+0x23/0x50
> [ 2038.726940] [<ffffffff8106d67f>] ? capable_wrt_inode_uidgid+0x6f/0x80
> [ 2038.726949] [<ffffffffa049a3bd>] ? xfs_vn_unlink+0x4d/0xa0 [xfs]
> [ 2038.726954] [<ffffffff811a54dd>] ? vfs_unlink+0x10d/0x180
> [ 2038.726960] [<ffffffffa03ecb24>] ? nfsd_unlink+0x184/0x240 [nfsd]
> [ 2038.726966] [<ffffffffa03f46ee>] ? nfsd3_proc_remove+0x7e/0x120 [nfsd]
> [ 2038.726972] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
> [ 2038.726978] [<ffffffffa034ad64>] ? svc_process_common+0x354/0x690
> [sunrpc]
> [ 2038.726982] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.726988] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 [sunrpc]
> [ 2038.726992] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd]
> [ 2038.726997] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.727001] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.727004] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.727007] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.727011] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.727014] INFO: task nfsd:2746 blocked for more than 120 seconds.
> [ 2038.727105] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.727181] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.727296] nfsd D ffff88003fc14340 0 2746 2
> 0x00000000
> [ 2038.727300] ffff88002f03b2f0 0000000000000046 ffff880039d92bb0
> ffff88002f2502d0
> [ 2038.727303] 0000000000014340 ffff88003c795fd8 0000000000014340
> ffff88002f03b2f0
> [ 2038.727306] ffff88003cec7380 ffff88003b6b99f0 7fffffffffffffff
> 7fffffffffffffff
> [ 2038.727309] Call Trace:
> [ 2038.727314] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
> [ 2038.727449] [<ffffffffa00bc7ab>] ? vmxnet3_xmit_frame+0x78b/0xad0
> [vmxnet3]
> [ 2038.727453] [<ffffffff814f193c>] ? __down_common+0x97/0xea
> [ 2038.727458] [<ffffffff81401e00>] ? dev_hard_start_xmit+0x2a0/0x5f0
> [ 2038.727467] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.727470] [<ffffffff810aa697>] ? down+0x37/0x40
> [ 2038.727478] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs]
> [ 2038.727486] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 [xfs]
> [ 2038.727494] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs]
> [ 2038.727502] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs]
> [ 2038.727509] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 [xfs]
> [ 2038.727519] [<ffffffffa04f11da>] ? xfs_trans_read_buf_map+0x34a/0x4f0
> [xfs]
> [ 2038.727529] [<ffffffffa04d5526>] ? xfs_read_agi+0x96/0x120 [xfs]
> [ 2038.727538] [<ffffffffa04e86db>] ? xlog_grant_head_check+0x4b/0xf0 [xfs]
> [ 2038.727548] [<ffffffffa04da400>] ? xfs_iunlink+0x50/0x180 [xfs]
> [ 2038.727552] [<ffffffff810689ed>] ? current_fs_time+0xd/0x50
> [ 2038.727561] [<ffffffffa04f1b2f>] ? xfs_trans_ichgtime+0x1f/0xa0 [xfs]
> [ 2038.727571] [<ffffffffa04a51ee>] ? kmem_zone_alloc+0x6e/0xf0 [xfs]
> [ 2038.727581] [<ffffffffa04f1b2f>] ? xfs_trans_ichgtime+0x1f/0xa0 [xfs]
> [ 2038.727590] [<ffffffffa04dada2>] ? xfs_remove+0x272/0x410 [xfs]
> [ 2038.727594] [<ffffffff811ad4f0>] ? d_lru_del+0x90/0x90
> [ 2038.727597] [<ffffffff811ada3c>] ? d_walk+0x6c/0x280
> [ 2038.727601] [<ffffffff811ad4f0>] ? d_lru_del+0x90/0x90
> [ 2038.727609] [<ffffffffa049a3bd>] ? xfs_vn_unlink+0x4d/0xa0 [xfs]
> [ 2038.727614] [<ffffffff811a58b4>] ? vfs_rmdir+0xa4/0x100
> [ 2038.727620] [<ffffffffa03ecbb3>] ? nfsd_unlink+0x213/0x240 [nfsd]
> [ 2038.727626] [<ffffffffa03f45ce>] ? nfsd3_proc_rmdir+0x7e/0x120 [nfsd]
> [ 2038.727631] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
> [ 2038.727638] [<ffffffffa034ad64>] ? svc_process_common+0x354/0x690
> [sunrpc]
> [ 2038.727643] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.727648] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 [sunrpc]
> [ 2038.727653] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd]
> [ 2038.727657] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.727661] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.727664] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.727668] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.727672] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.727675] INFO: task nfsd:2747 blocked for more than 120 seconds.
> [ 2038.727766] Not tainted 3.14-0.bpo.1-amd64 #1
> [ 2038.727846] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 2038.727966] nfsd D ffff88003fc14340 0 2747 2
> 0x00000000
> [ 2038.727970] ffff88003d1d0450 0000000000000046 0000000000000001
> ffff88002f2502d0
> [ 2038.727973] 0000000000014340 ffff88003c7d7fd8 0000000000014340
> ffff88003d1d0450
> [ 2038.727976] 0000000000000001 ffff88003fc14bd0 ffff88003d1d0450
> ffffffff8112d400
> [ 2038.727979] Call Trace:
> [ 2038.727984] [<ffffffff8112d400>] ? __lock_page+0x70/0x70
> [ 2038.727989] [<ffffffff814efb97>] ? io_schedule+0x87/0xd0
> [ 2038.727992] [<ffffffff8112d409>] ? sleep_on_page+0x9/0x10
> [ 2038.727996] [<ffffffff814f0182>] ? __wait_on_bit+0x52/0x80
> [ 2038.728000] [<ffffffff8112e0f7>] ? find_get_pages_tag+0xc7/0x180
> [ 2038.728004] [<ffffffff8112d543>] ? wait_on_page_bit+0x73/0x80
> [ 2038.728008] [<ffffffff810a6830>] ? wake_atomic_t_function+0x30/0x30
> [ 2038.728011] [<ffffffff8112d624>] ? filemap_fdatawait_range+0xd4/0x150
> [ 2038.728015] [<ffffffff8112ec18>] ?
> filemap_write_and_wait_range+0x48/0x90
> [ 2038.728024] [<ffffffffa049b6e3>] ? xfs_setattr_size+0x2d3/0x3b0 [xfs]
> [ 2038.728028] [<ffffffff814f1a19>] ? down_write+0x9/0x30
> [ 2038.728038] [<ffffffffa04d7608>] ? xfs_ilock+0xd8/0x160 [xfs]
> [ 2038.728047] [<ffffffffa049b818>] ? xfs_vn_setattr+0x58/0x70 [xfs]
> [ 2038.728051] [<ffffffff811b309b>] ? notify_change+0x1eb/0x3d0
> [ 2038.728057] [<ffffffffa03ed5fc>] ? nfsd_setattr+0x24c/0x4b0 [nfsd]
> [ 2038.728064] [<ffffffffa03f5653>] ? nfsd3_proc_setattr+0x73/0xc0 [nfsd]
> [ 2038.728068] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
> [ 2038.728075] [<ffffffffa034ad64>] ? svc_process_common+0x354/0x690
> [sunrpc]
> [ 2038.728080] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
> [ 2038.728086] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 [sunrpc]
> [ 2038.728091] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd]
> [ 2038.728095] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd]
> [ 2038.728098] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
> [ 2038.728102] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2038.728105] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
> [ 2038.728109] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
> [ 2676.876333] [sched_delayed] sched: RT throttling activated
Well, these don't point at rbd at all. Are you seeing *any* progress
when this happens? Could it be that things just get very slow and
don't actually hang?
Can you try watching sysfs osdc file for a while to see if requests are
going through or not? (/sys/kernel/debug/ceph/<fsid>.<id>/osdc)
Thanks,
Ilya
