Ilya, Was wondering if you've had a chance to look into performance issues with rbd and the patched kernel? I've downloaded 3.16.3 and running some dd tests, which were producing hang tasks in the past. I've noticed that i can't get past 20mb/s on the rbd mounted volume. I am sure I was hitting over 60MB/s before. Cheers andrei ----- Original Message ----- > From: "Ilya Dryomov" <ilya.dryomov at inktank.com> > To: "Micha Krause" <micha at krausam.de> > Cc: ceph-users at lists.ceph.com > Sent: Wednesday, 24 September, 2014 3:33:20 PM > Subject: Re: Frequent Crashes on rbd to nfs gateway > Server > On Wed, Sep 24, 2014 at 4:52 PM, Micha Krause <micha at krausam.de> > wrote: > > Hi, > > > >> Like I mentioned in my other reply, I'd be very interested in any > >> > >> similar messages on kernel other than 3.15.*, 3.16.1 and 3.16.2. > >> One > >> hung task stack trace is usually not enough to diagnose this sort > >> of > >> problems. > > > > > > Ok, here is a more complete dmesg output from kernel 3.14: > > > > [ 22.250600] rbd: loaded > > [ 23.159914] libceph: client24407525 fsid > > 46e857ee-855c-4165-8413-8950f8d081be > > [ 23.289691] libceph: mon1 10.210.34.11:6789 session established > > [ 23.890625] rbd2: unknown partition table > > [ 23.890702] rbd: rbd2: added with size 0x10000000000 > > [ 23.937051] rbd0: unknown partition table > > [ 23.937144] rbd: rbd0: added with size 0x10000000000 > > [ 24.052402] rbd1: unknown partition table > > [ 24.052479] rbd: rbd1: added with size 0xa0000000000 > > [ 24.396333] rbd3: unknown partition table > > [ 24.396430] rbd: rbd3: added with size 0x10000000000 > > [ 25.927373] SGI XFS with ACLs, security attributes, realtime, > > large > > block/inode numbers, no debug enabled > > [ 25.960975] XFS (rbd1): Mounting Filesystem > > [ 25.961072] XFS (rbd3): Mounting Filesystem > > [ 25.961637] XFS (rbd2): Mounting Filesystem > > [ 25.961708] XFS (rbd0): Mounting Filesystem > > [ 28.236952] XFS (rbd3): Starting recovery (logdev: internal) > > [ 28.794631] XFS (rbd1): Starting recovery (logdev: internal) > > [ 31.501516] XFS (rbd0): Starting recovery (logdev: internal) > > [ 35.498950] XFS (rbd2): Starting recovery (logdev: internal) > > [ 63.601465] XFS (rbd0): Ending recovery (logdev: internal) > > [ 64.214852] XFS (rbd3): Ending recovery (logdev: internal) > > [ 64.783531] rbd4: unknown partition table > > [ 64.784005] rbd: rbd4: added with size 0x10000000000 > > [ 65.280960] XFS (rbd4): Mounting Filesystem > > [ 68.443439] XFS (rbd2): Ending recovery (logdev: internal) > > [ 69.030358] XFS (rbd4): Starting recovery (logdev: internal) > > [ 69.945523] rbd5: unknown partition table > > [ 69.946021] rbd: rbd5: added with size 0x10000000000 > > [ 70.398567] XFS (rbd5): Mounting Filesystem > > [ 71.187934] XFS (rbd5): Starting recovery (logdev: internal) > > [ 74.144173] rbd6: unknown partition table > > [ 74.144630] rbd: rbd6: added with size 0x10000000000 > > [ 75.402767] XFS (rbd6): Mounting Filesystem > > [ 76.133654] XFS (rbd6): Starting recovery (logdev: internal) > > [ 111.131893] XFS (rbd4): Ending recovery (logdev: internal) > > [ 112.460383] rbd7: unknown partition table > > [ 112.460898] rbd: rbd7: added with size 0x10000000000 > > [ 116.834457] XFS (rbd5): Ending recovery (logdev: internal) > > [ 116.949218] XFS (rbd6): Ending recovery (logdev: internal) > > [ 166.357039] XFS (rbd1): Ending recovery (logdev: internal) > > [ 167.531353] XFS (rbd7): Mounting Filesystem > > [ 168.303166] XFS (rbd7): Starting recovery (logdev: internal) > > [ 172.477811] XFS (rbd7): Ending recovery (logdev: internal) > > [ 2038.723394] INFO: task kthreadd:2 blocked for more than 120 > > seconds. > > [ 2038.723497] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.723553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.723637] kthreadd D ffff88003fc14340 0 2 0 > > 0x00000000 > > [ 2038.723641] ffff88003fa3a8e0 0000000000000046 ffff88003fa43628 > > ffffffff81813480 > > [ 2038.723644] 0000000000014340 ffff88003fa43fd8 0000000000014340 > > ffff88003fa3a8e0 > > [ 2038.723646] ffff88003fa43638 ffff8800006a7410 7fffffffffffffff > > ffff8800006a7408 > > [ 2038.723648] Call Trace: > > [ 2038.723660] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.723665] [<ffffffff8127300b>] ? blk_finish_plug+0xb/0x30 > > [ 2038.723677] [<ffffffffa048c957>] ? _xfs_buf_ioapply+0x277/0x2e0 > > [xfs] > > [ 2038.723680] [<ffffffff814f0b94>] ? > > wait_for_completion+0xa4/0x110 > > [ 2038.723685] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.723691] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs] > > [ 2038.723696] [<ffffffffa048cf56>] ? xfs_buf_iowait+0x96/0xf0 > > [xfs] > > [ 2038.723703] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs] > > [ 2038.723711] [<ffffffffa0494b24>] ? xfs_reclaim_inode+0x2f4/0x310 > > [xfs] > > [ 2038.723720] [<ffffffffa0494d37>] ? > > xfs_reclaim_inodes_ag+0x1f7/0x320 > > [xfs] > > [ 2038.723729] [<ffffffffa04959dc>] ? > > xfs_reclaim_inodes_nr+0x2c/0x40 [xfs] > > [ 2038.723736] [<ffffffff8119b9f7>] ? super_cache_scan+0x167/0x170 > > [ 2038.723742] [<ffffffff8113e406>] ? shrink_slab_node+0x126/0x290 > > [ 2038.723746] [<ffffffff811923b3>] ? vmpressure+0x23/0xa0 > > [ 2038.723750] [<ffffffff81140862>] ? shrink_slab+0x82/0x130 > > [ 2038.723755] [<ffffffff81143722>] ? > > do_try_to_free_pages+0x3c2/0x510 > > [ 2038.723762] [<ffffffff81136cff>] ? > > get_page_from_freelist+0x59f/0x8a0 > > [ 2038.723766] [<ffffffff81143bd7>] ? try_to_free_pages+0x107/0x1f0 > > [ 2038.723770] [<ffffffff811376f0>] ? > > __alloc_pages_nodemask+0x6f0/0xaf0 > > [ 2038.723776] [<ffffffff810624eb>] ? copy_process+0x20b/0x1ba0 > > [ 2038.723782] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.723788] [<ffffffff8109dea1>] ? dequeue_task_fair+0x231/0x7f0 > > [ 2038.723795] [<ffffffff810145af>] ? __switch_to+0x12f/0x4e0 > > [ 2038.723798] [<ffffffff8109a745>] ? set_next_entity+0x35/0x80 > > [ 2038.723802] [<ffffffff81063fad>] ? do_fork+0x6d/0x340 > > [ 2038.723805] [<ffffffff810642a1>] ? kernel_thread+0x21/0x30 > > [ 2038.723809] [<ffffffff810872e5>] ? kthreadd+0x135/0x190 > > [ 2038.723813] [<ffffffff810871b0>] ? > > kthread_create_on_cpu+0x60/0x60 > > [ 2038.723817] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.723821] [<ffffffff810871b0>] ? > > kthread_create_on_cpu+0x60/0x60 > > [ 2038.723828] INFO: task kswapd0:23 blocked for more than 120 > > seconds. > > [ 2038.723918] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.724002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.724111] kswapd0 D ffff88003fc14340 0 23 2 > > 0x00000000 > > [ 2038.724115] ffff88003bb8aaa0 0000000000000046 0000003300000000 > > ffff88002f2502d0 > > [ 2038.724117] 0000000000014340 ffff88003bb91fd8 0000000000014340 > > ffff88003bb8aaa0 > > [ 2038.724118] 0000000000000000 ffff88003bb91610 7fffffffffffffff > > ffff88003bb91608 > > [ 2038.724120] Call Trace: > > [ 2038.724125] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.724134] [<ffffffffa04dc6b8>] ? > > xfs_iext_bno_to_ext+0x88/0x160 [xfs] > > [ 2038.724137] [<ffffffff810968fb>] ? try_to_wake_up+0xcb/0x280 > > [ 2038.724140] [<ffffffff814f0b94>] ? > > wait_for_completion+0xa4/0x110 > > [ 2038.724143] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.724148] [<ffffffffa0488eed>] ? xfs_bmapi_allocate+0x8d/0xb0 > > [xfs] > > [ 2038.724156] [<ffffffffa04bafa8>] ? xfs_bmapi_write+0x458/0x770 > > [xfs] > > [ 2038.724160] [<ffffffff8107e86b>] ? __queue_work+0x14b/0x3b0 > > [ 2038.724165] [<ffffffffa04883f0>] ? > > xfs_bmap_count_tree+0x1f0/0x1f0 [xfs] > > [ 2038.724171] [<ffffffffa0499b38>] ? > > xfs_iomap_write_allocate+0x148/0x360 > > [xfs] > > [ 2038.724176] [<ffffffffa0485025>] ? xfs_map_blocks+0x195/0x270 > > [xfs] > > [ 2038.724181] [<ffffffffa04862df>] ? xfs_vm_writepage+0x20f/0x570 > > [xfs] > > [ 2038.724184] [<ffffffff811647d0>] ? > > __page_check_address+0x1f0/0x1f0 > > [ 2038.724187] [<ffffffff811418a0>] ? shrink_page_list+0x790/0xa60 > > [ 2038.724194] [<ffffffffa04e5753>] ? xfs_perag_get_tag+0x43/0x100 > > [xfs] > > [ 2038.724197] [<ffffffff811421dd>] ? > > shrink_inactive_list+0x19d/0x510 > > [ 2038.724200] [<ffffffff81142c00>] ? shrink_lruvec+0x330/0x630 > > [ 2038.724203] [<ffffffff81142f6e>] ? shrink_zone+0x6e/0x1a0 > > [ 2038.724205] [<ffffffff811441ab>] ? balance_pgdat+0x38b/0x5c0 > > [ 2038.724208] [<ffffffff81070397>] ? > > try_to_del_timer_sync+0x47/0x60 > > [ 2038.724211] [<ffffffff81144547>] ? kswapd+0x167/0x460 > > [ 2038.724214] [<ffffffff810a67d0>] ? __wake_up_sync+0x10/0x10 > > [ 2038.724217] [<ffffffff811443e0>] ? balance_pgdat+0x5c0/0x5c0 > > [ 2038.724219] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.724221] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.724223] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.724226] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.724244] INFO: task nfsd:2740 blocked for more than 120 > > seconds. > > [ 2038.724317] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.724373] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.724456] nfsd D ffff88003fc14340 0 2740 2 > > 0x00000000 > > [ 2038.724459] ffff88003b794350 0000000000000046 ffff88003b9a5668 > > ffff88002f2502d0 > > [ 2038.724461] 0000000000014340 ffff88003b9a5fd8 0000000000014340 > > ffff88003b794350 > > [ 2038.724463] ffff88003b9a5678 ffff880009997610 7fffffffffffffff > > ffff880009997608 > > [ 2038.724465] Call Trace: > > [ 2038.724468] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.724471] [<ffffffff8127300b>] ? blk_finish_plug+0xb/0x30 > > [ 2038.724476] [<ffffffffa048c957>] ? _xfs_buf_ioapply+0x277/0x2e0 > > [xfs] > > [ 2038.724479] [<ffffffff814f0b94>] ? > > wait_for_completion+0xa4/0x110 > > [ 2038.724482] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.724487] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs] > > [ 2038.724492] [<ffffffffa048cf56>] ? xfs_buf_iowait+0x96/0xf0 > > [xfs] > > [ 2038.724498] [<ffffffffa048cfd3>] ? xfs_bwrite+0x23/0x60 [xfs] > > [ 2038.724503] [<ffffffffa0494b24>] ? xfs_reclaim_inode+0x2f4/0x310 > > [xfs] > > [ 2038.724509] [<ffffffffa0494d37>] ? > > xfs_reclaim_inodes_ag+0x1f7/0x320 > > [xfs] > > [ 2038.724515] [<ffffffffa04959dc>] ? > > xfs_reclaim_inodes_nr+0x2c/0x40 [xfs] > > [ 2038.724518] [<ffffffff8119b9f7>] ? super_cache_scan+0x167/0x170 > > [ 2038.724520] [<ffffffff8113e406>] ? shrink_slab_node+0x126/0x290 > > [ 2038.724523] [<ffffffff811923b3>] ? vmpressure+0x23/0xa0 > > [ 2038.724525] [<ffffffff81140862>] ? shrink_slab+0x82/0x130 > > [ 2038.724527] [<ffffffff81143722>] ? > > do_try_to_free_pages+0x3c2/0x510 > > [ 2038.724530] [<ffffffff81143bd7>] ? try_to_free_pages+0x107/0x1f0 > > [ 2038.724533] [<ffffffff811376f0>] ? > > __alloc_pages_nodemask+0x6f0/0xaf0 > > [ 2038.724537] [<ffffffff81176365>] ? > > alloc_pages_current+0xb5/0x180 > > [ 2038.724548] [<ffffffffa035a0bc>] ? svc_recv+0xbc/0xa10 [sunrpc] > > [ 2038.724552] [<ffffffffa0358f65>] ? svc_xprt_put+0x5/0x20 > > [sunrpc] > > [ 2038.724557] [<ffffffffa035aae0>] ? svc_send+0xd0/0x100 [sunrpc] > > [ 2038.724563] [<ffffffffa03e66c5>] ? nfsd+0xa5/0x130 [nfsd] > > [ 2038.724566] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.724569] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.724571] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.724573] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.724575] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.724577] INFO: task nfsd:2741 blocked for more than 120 > > seconds. > > [ 2038.724641] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.724696] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.724819] nfsd D ffff88003fc14340 0 2741 2 > > 0x00000000 > > [ 2038.724823] ffff88003b794c20 0000000000000046 0000000000000008 > > ffff8800176f4b60 > > [ 2038.724826] 0000000000014340 ffff88003b9e7fd8 0000000000014340 > > ffff88003b794c20 > > [ 2038.724828] 00000000000006f4 ffff88003b794c20 fffffffe00000001 > > ffff88000a3df890 > > [ 2038.724831] Call Trace: > > [ 2038.724835] [<ffffffff814f2145>] ? > > rwsem_down_write_failed+0x105/0x1c0 > > [ 2038.724839] [<ffffffff812a53e3>] ? > > call_rwsem_down_write_failed+0x13/0x20 > > [ 2038.724841] [<ffffffff814f1a34>] ? down_write+0x24/0x30 > > [ 2038.724848] [<ffffffffa049b012>] ? > > xfs_setattr_nonsize+0x1c2/0x5c0 [xfs] > > [ 2038.724855] [<ffffffffa049b7eb>] ? xfs_vn_setattr+0x2b/0x70 > > [xfs] > > [ 2038.724858] [<ffffffff811b309b>] ? notify_change+0x1eb/0x3d0 > > [ 2038.724863] [<ffffffffa03ed5fc>] ? nfsd_setattr+0x24c/0x4b0 > > [nfsd] > > [ 2038.724867] [<ffffffffa03f5653>] ? nfsd3_proc_setattr+0x73/0xc0 > > [nfsd] > > [ 2038.724871] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 > > [nfsd] > > [ 2038.724876] [<ffffffffa034ad64>] ? > > svc_process_common+0x354/0x690 > > [sunrpc] > > [ 2038.724879] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.724883] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 > > [sunrpc] > > [ 2038.724886] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd] > > [ 2038.724890] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.724892] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.724894] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.724896] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.724898] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.724900] INFO: task nfsd:2742 blocked for more than 120 > > seconds. > > [ 2038.724977] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.725221] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.725314] nfsd D ffff88003fc14340 0 2742 2 > > 0x00000000 > > [ 2038.725316] ffff88003b7162d0 0000000000000046 ffff880025460060 > > ffff88002f2502d0 > > [ 2038.725318] 0000000000014340 ffff88003c709fd8 0000000000014340 > > ffff88003b7162d0 > > [ 2038.725320] ffff88003c709810 ffff88000dab4370 7fffffffffffffff > > 7fffffffffffffff > > [ 2038.725322] Call Trace: > > [ 2038.725326] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.725333] [<ffffffffa04b150a>] ? > > xfs_bmap_search_extents+0x6a/0xf0 > > [xfs] > > [ 2038.725336] [<ffffffff814f193c>] ? __down_common+0x97/0xea > > [ 2038.725341] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.725344] [<ffffffff810aa697>] ? down+0x37/0x40 > > [ 2038.725349] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs] > > [ 2038.725354] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.725360] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 > > [xfs] > > [ 2038.725365] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 > > [xfs] > > [ 2038.725368] [<ffffffff814f2719>] ? > > _raw_spin_unlock_irqrestore+0x9/0x10 > > [ 2038.725375] [<ffffffffa04f1112>] ? > > xfs_trans_read_buf_map+0x282/0x4f0 > > [xfs] > > [ 2038.725382] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 > > [xfs] > > [ 2038.725386] [<ffffffff81401e9e>] ? > > dev_hard_start_xmit+0x33e/0x5f0 > > [ 2038.725393] [<ffffffffa04c95ef>] ? xfs_dir3_block_read+0x2f/0x70 > > [xfs] > > [ 2038.725400] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs] > > [ 2038.725406] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs] > > [ 2038.725413] [<ffffffffa04c9668>] ? > > xfs_dir2_block_lookup_int+0x38/0x1f0 > > [xfs] > > [ 2038.725419] [<ffffffffa049530e>] ? xfs_iget+0x4ae/0x810 [xfs] > > [ 2038.725425] [<ffffffffa04ca094>] ? > > xfs_dir2_block_lookup+0x24/0x120 > > [xfs] > > [ 2038.725432] [<ffffffffa04c86c0>] ? xfs_dir2_isblock+0x20/0x60 > > [xfs] > > [ 2038.725438] [<ffffffffa04c8cd0>] ? xfs_dir_lookup+0xf0/0x1a0 > > [xfs] > > [ 2038.725445] [<ffffffffa04d8790>] ? xfs_lookup+0xe0/0x140 [xfs] > > [ 2038.725448] [<ffffffff811af96a>] ? __d_alloc+0x13a/0x180 > > [ 2038.725454] [<ffffffffa049a120>] ? xfs_vn_lookup+0x50/0x90 [xfs] > > [ 2038.725457] [<ffffffff811a2299>] ? generic_permission+0xf9/0x1a0 > > [ 2038.725459] [<ffffffff811a1c64>] ? lookup_real+0x14/0x50 > > [ 2038.725462] [<ffffffff811a2142>] ? __lookup_hash+0x32/0x50 > > [ 2038.725464] [<ffffffff811a65dd>] ? lookup_one_len+0xbd/0x110 > > [ 2038.725469] [<ffffffffa03eb3e7>] ? > > nfsd_lookup_dentry+0x127/0x490 [nfsd] > > [ 2038.725474] [<ffffffffa03eb7cb>] ? nfsd_lookup+0x7b/0x160 [nfsd] > > [ 2038.725481] [<ffffffffa034e3e0>] ? > > svcauth_null_release+0x60/0x60 > > [sunrpc] > > [ 2038.725487] [<ffffffffa03f5531>] ? nfsd3_proc_lookup+0xd1/0x180 > > [nfsd] > > [ 2038.725491] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 > > [nfsd] > > [ 2038.725498] [<ffffffffa034ad64>] ? > > svc_process_common+0x354/0x690 > > [sunrpc] > > [ 2038.725502] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.725507] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 > > [sunrpc] > > [ 2038.725511] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd] > > [ 2038.725514] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.725517] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.725519] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.725521] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.725524] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.725526] INFO: task nfsd:2743 blocked for more than 120 > > seconds. > > [ 2038.725589] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.725643] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.725735] nfsd D ffff88003fc14340 0 2743 2 > > 0x00000000 > > [ 2038.725739] ffff88003d468a60 0000000000000046 0000000000002000 > > ffffffff81813480 > > [ 2038.725742] 0000000000014340 ffff88003c72dfd8 0000000000014340 > > ffff88003d468a60 > > [ 2038.725744] ffff880037acd1c0 ffff880039d78870 7fffffffffffffff > > 7fffffffffffffff > > [ 2038.725747] Call Trace: > > [ 2038.725752] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.725755] [<ffffffff814f193c>] ? __down_common+0x97/0xea > > [ 2038.725763] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.725768] [<ffffffff810aa697>] ? down+0x37/0x40 > > [ 2038.725775] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs] > > [ 2038.725782] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.725790] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 > > [xfs] > > [ 2038.725797] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 > > [xfs] > > [ 2038.725801] [<ffffffff814f2719>] ? > > _raw_spin_unlock_irqrestore+0x9/0x10 > > [ 2038.725811] [<ffffffffa04f1112>] ? > > xfs_trans_read_buf_map+0x282/0x4f0 > > [xfs] > > [ 2038.725821] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 > > [xfs] > > [ 2038.725824] [<ffffffff81070108>] ? internal_add_timer+0x18/0x50 > > [ 2038.725831] [<ffffffffa04c95ef>] ? xfs_dir3_block_read+0x2f/0x70 > > [xfs] > > [ 2038.725838] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs] > > [ 2038.725843] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs] > > [ 2038.725850] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs] > > [ 2038.725855] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs] > > [ 2038.725861] [<ffffffffa04c9668>] ? > > xfs_dir2_block_lookup_int+0x38/0x1f0 > > [xfs] > > [ 2038.725867] [<ffffffffa049530e>] ? xfs_iget+0x4ae/0x810 [xfs] > > [ 2038.725873] [<ffffffffa04ca094>] ? > > xfs_dir2_block_lookup+0x24/0x120 > > [xfs] > > [ 2038.725879] [<ffffffffa04c86c0>] ? xfs_dir2_isblock+0x20/0x60 > > [xfs] > > [ 2038.725885] [<ffffffffa04c8cd0>] ? xfs_dir_lookup+0xf0/0x1a0 > > [xfs] > > [ 2038.725892] [<ffffffffa04d8790>] ? xfs_lookup+0xe0/0x140 [xfs] > > [ 2038.725894] [<ffffffff811af96a>] ? __d_alloc+0x13a/0x180 > > [ 2038.725901] [<ffffffffa049a120>] ? xfs_vn_lookup+0x50/0x90 [xfs] > > [ 2038.725903] [<ffffffff811a1c64>] ? lookup_real+0x14/0x50 > > [ 2038.725905] [<ffffffff811a2142>] ? __lookup_hash+0x32/0x50 > > [ 2038.725908] [<ffffffff811a65dd>] ? lookup_one_len+0xbd/0x110 > > [ 2038.725912] [<ffffffffa03eb3e7>] ? > > nfsd_lookup_dentry+0x127/0x490 [nfsd] > > [ 2038.725916] [<ffffffffa03eb7cb>] ? nfsd_lookup+0x7b/0x160 [nfsd] > > [ 2038.725920] [<ffffffffa034e3e0>] ? > > svcauth_null_release+0x60/0x60 > > [sunrpc] > > [ 2038.725924] [<ffffffffa03f5531>] ? nfsd3_proc_lookup+0xd1/0x180 > > [nfsd] > > [ 2038.725928] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 > > [nfsd] > > [ 2038.725932] [<ffffffffa034ad64>] ? > > svc_process_common+0x354/0x690 > > [sunrpc] > > [ 2038.725935] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.725939] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 > > [sunrpc] > > [ 2038.725944] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd] > > [ 2038.725949] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.725952] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.725956] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.725959] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.725963] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.725966] INFO: task nfsd:2744 blocked for more than 120 > > seconds. > > [ 2038.726054] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.726125] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.726244] nfsd D ffff88003fc14340 0 2744 2 > > 0x00000000 > > [ 2038.726248] ffff88003d1d0d20 0000000000000046 ffff88001f3849a8 > > ffff88002f2502d0 > > [ 2038.726251] 0000000000014340 ffff88003c74ffd8 0000000000014340 > > ffff88003d1d0d20 > > [ 2038.726253] 0000000000000001 ffff88001be09970 7fffffffffffffff > > 7fffffffffffffff > > [ 2038.726256] Call Trace: > > [ 2038.726261] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.726266] [<ffffffff814f193c>] ? __down_common+0x97/0xea > > [ 2038.726274] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.726278] [<ffffffff810aa697>] ? down+0x37/0x40 > > [ 2038.726285] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs] > > [ 2038.726293] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.726300] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 > > [xfs] > > [ 2038.726308] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 > > [xfs] > > [ 2038.726318] [<ffffffffa04f1112>] ? > > xfs_trans_read_buf_map+0x282/0x4f0 > > [xfs] > > [ 2038.726327] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 > > [xfs] > > [ 2038.726337] [<ffffffffa04e5669>] ? xfs_perag_get+0x39/0xe0 [xfs] > > [ 2038.726345] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs] > > [ 2038.726354] [<ffffffffa04cc4ee>] ? > > xfs_dir3_leaf_read.constprop.13+0x2e/0x80 [xfs] > > [ 2038.726357] [<ffffffff8114d31d>] ? zone_statistics+0x9d/0xa0 > > [ 2038.726366] [<ffffffffa04cd5df>] ? > > xfs_dir2_leaf_lookup_int+0x4f/0x2d0 > > [xfs] > > [ 2038.726373] [<ffffffffa049530e>] ? xfs_iget+0x4ae/0x810 [xfs] > > [ 2038.726382] [<ffffffffa04cde39>] ? > > xfs_dir2_leaf_lookup+0x29/0x140 [xfs] > > [ 2038.726390] [<ffffffffa04c8720>] ? xfs_dir2_isleaf+0x20/0x60 > > [xfs] > > [ 2038.726396] [<ffffffffa03e9a20>] ? > > _fh_update.isra.9.part.10+0x50/0x50 > > [nfsd] > > [ 2038.726404] [<ffffffffa04c8d6a>] ? xfs_dir_lookup+0x18a/0x1a0 > > [xfs] > > [ 2038.726413] [<ffffffffa04d8790>] ? xfs_lookup+0xe0/0x140 [xfs] > > [ 2038.726417] [<ffffffff811af96a>] ? __d_alloc+0x13a/0x180 > > [ 2038.726425] [<ffffffffa049a120>] ? xfs_vn_lookup+0x50/0x90 [xfs] > > [ 2038.726428] [<ffffffff811a1c64>] ? lookup_real+0x14/0x50 > > [ 2038.726432] [<ffffffff811a2142>] ? __lookup_hash+0x32/0x50 > > [ 2038.726435] [<ffffffff811a65dd>] ? lookup_one_len+0xbd/0x110 > > [ 2038.726441] [<ffffffffa03eda76>] ? do_nfsd_create+0x216/0x610 > > [nfsd] > > [ 2038.726446] [<ffffffffa03f4f4d>] ? nfsd3_proc_create+0x16d/0x250 > > [nfsd] > > [ 2038.726450] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 > > [nfsd] > > [ 2038.726457] [<ffffffffa034ad64>] ? > > svc_process_common+0x354/0x690 > > [sunrpc] > > [ 2038.726461] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.726466] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 > > [sunrpc] > > [ 2038.726470] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd] > > [ 2038.726475] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.726478] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.726481] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.726485] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.726488] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.726491] INFO: task nfsd:2745 blocked for more than 120 > > seconds. > > [ 2038.726579] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.726657] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.726773] nfsd D ffff88003fc14340 0 2745 2 > > 0x00000000 > > [ 2038.726777] ffff88003d1e0da0 0000000000000046 ffff880005900800 > > ffff88002f2502d0 > > [ 2038.726780] 0000000000014340 ffff88003c773fd8 0000000000014340 > > ffff88003d1e0da0 > > [ 2038.726782] 0000000100000003 ffff8800063930f0 7fffffffffffffff > > 7fffffffffffffff > > [ 2038.726785] Call Trace: > > [ 2038.726790] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.726794] [<ffffffff814f193c>] ? __down_common+0x97/0xea > > [ 2038.726802] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.726805] [<ffffffff810aa697>] ? down+0x37/0x40 > > [ 2038.726812] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs] > > [ 2038.726820] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.726826] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 > > [xfs] > > [ 2038.726834] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 > > [xfs] > > [ 2038.726844] [<ffffffffa04f11da>] ? > > xfs_trans_read_buf_map+0x34a/0x4f0 > > [xfs] > > [ 2038.726853] [<ffffffffa04c560e>] ? xfs_da_read_buf+0xfe/0x280 > > [xfs] > > [ 2038.726862] [<ffffffffa04c57a9>] ? xfs_da3_node_read+0x19/0xc0 > > [xfs] > > [ 2038.726868] [<ffffffff8142163c>] ? sch_direct_xmit+0x7c/0x1d0 > > [ 2038.726878] [<ffffffffa04c6d1e>] ? > > xfs_da3_node_lookup_int+0x5e/0x2f0 > > [xfs] > > [ 2038.726887] [<ffffffffa04d1777>] ? > > xfs_dir2_node_removename+0x77/0x6d0 > > [xfs] > > [ 2038.726894] [<ffffffffa0354f2c>] ? cache_check+0x5c/0x360 > > [sunrpc] > > [ 2038.726903] [<ffffffffa04b8fee>] ? > > xfs_bmap_last_extent+0x6e/0x90 [xfs] > > [ 2038.726912] [<ffffffffa04b9127>] ? > > xfs_bmap_last_offset+0x87/0xc0 [xfs] > > [ 2038.726920] [<ffffffffa04c8bd5>] ? > > xfs_dir_removename+0x185/0x190 [xfs] > > [ 2038.726930] [<ffffffffa04dadf3>] ? xfs_remove+0x2c3/0x410 [xfs] > > [ 2038.726936] [<ffffffff8106d5e3>] ? ns_capable+0x23/0x50 > > [ 2038.726940] [<ffffffff8106d67f>] ? > > capable_wrt_inode_uidgid+0x6f/0x80 > > [ 2038.726949] [<ffffffffa049a3bd>] ? xfs_vn_unlink+0x4d/0xa0 [xfs] > > [ 2038.726954] [<ffffffff811a54dd>] ? vfs_unlink+0x10d/0x180 > > [ 2038.726960] [<ffffffffa03ecb24>] ? nfsd_unlink+0x184/0x240 > > [nfsd] > > [ 2038.726966] [<ffffffffa03f46ee>] ? nfsd3_proc_remove+0x7e/0x120 > > [nfsd] > > [ 2038.726972] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 > > [nfsd] > > [ 2038.726978] [<ffffffffa034ad64>] ? > > svc_process_common+0x354/0x690 > > [sunrpc] > > [ 2038.726982] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.726988] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 > > [sunrpc] > > [ 2038.726992] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd] > > [ 2038.726997] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.727001] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.727004] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.727007] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.727011] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.727014] INFO: task nfsd:2746 blocked for more than 120 > > seconds. > > [ 2038.727105] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.727181] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.727296] nfsd D ffff88003fc14340 0 2746 2 > > 0x00000000 > > [ 2038.727300] ffff88002f03b2f0 0000000000000046 ffff880039d92bb0 > > ffff88002f2502d0 > > [ 2038.727303] 0000000000014340 ffff88003c795fd8 0000000000014340 > > ffff88002f03b2f0 > > [ 2038.727306] ffff88003cec7380 ffff88003b6b99f0 7fffffffffffffff > > 7fffffffffffffff > > [ 2038.727309] Call Trace: > > [ 2038.727314] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > > [ 2038.727449] [<ffffffffa00bc7ab>] ? > > vmxnet3_xmit_frame+0x78b/0xad0 > > [vmxnet3] > > [ 2038.727453] [<ffffffff814f193c>] ? __down_common+0x97/0xea > > [ 2038.727458] [<ffffffff81401e00>] ? > > dev_hard_start_xmit+0x2a0/0x5f0 > > [ 2038.727467] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.727470] [<ffffffff810aa697>] ? down+0x37/0x40 > > [ 2038.727478] [<ffffffffa048be02>] ? xfs_buf_lock+0x32/0xf0 [xfs] > > [ 2038.727486] [<ffffffffa048bfaa>] ? _xfs_buf_find+0xea/0x280 > > [xfs] > > [ 2038.727494] [<ffffffffa0494f07>] ? xfs_iget+0xa7/0x810 [xfs] > > [ 2038.727502] [<ffffffffa048c215>] ? xfs_buf_get_map+0x35/0x1a0 > > [xfs] > > [ 2038.727509] [<ffffffffa048d153>] ? xfs_buf_read_map+0x33/0x130 > > [xfs] > > [ 2038.727519] [<ffffffffa04f11da>] ? > > xfs_trans_read_buf_map+0x34a/0x4f0 > > [xfs] > > [ 2038.727529] [<ffffffffa04d5526>] ? xfs_read_agi+0x96/0x120 [xfs] > > [ 2038.727538] [<ffffffffa04e86db>] ? > > xlog_grant_head_check+0x4b/0xf0 [xfs] > > [ 2038.727548] [<ffffffffa04da400>] ? xfs_iunlink+0x50/0x180 [xfs] > > [ 2038.727552] [<ffffffff810689ed>] ? current_fs_time+0xd/0x50 > > [ 2038.727561] [<ffffffffa04f1b2f>] ? xfs_trans_ichgtime+0x1f/0xa0 > > [xfs] > > [ 2038.727571] [<ffffffffa04a51ee>] ? kmem_zone_alloc+0x6e/0xf0 > > [xfs] > > [ 2038.727581] [<ffffffffa04f1b2f>] ? xfs_trans_ichgtime+0x1f/0xa0 > > [xfs] > > [ 2038.727590] [<ffffffffa04dada2>] ? xfs_remove+0x272/0x410 [xfs] > > [ 2038.727594] [<ffffffff811ad4f0>] ? d_lru_del+0x90/0x90 > > [ 2038.727597] [<ffffffff811ada3c>] ? d_walk+0x6c/0x280 > > [ 2038.727601] [<ffffffff811ad4f0>] ? d_lru_del+0x90/0x90 > > [ 2038.727609] [<ffffffffa049a3bd>] ? xfs_vn_unlink+0x4d/0xa0 [xfs] > > [ 2038.727614] [<ffffffff811a58b4>] ? vfs_rmdir+0xa4/0x100 > > [ 2038.727620] [<ffffffffa03ecbb3>] ? nfsd_unlink+0x213/0x240 > > [nfsd] > > [ 2038.727626] [<ffffffffa03f45ce>] ? nfsd3_proc_rmdir+0x7e/0x120 > > [nfsd] > > [ 2038.727631] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 > > [nfsd] > > [ 2038.727638] [<ffffffffa034ad64>] ? > > svc_process_common+0x354/0x690 > > [sunrpc] > > [ 2038.727643] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.727648] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 > > [sunrpc] > > [ 2038.727653] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd] > > [ 2038.727657] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.727661] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.727664] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.727668] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.727672] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.727675] INFO: task nfsd:2747 blocked for more than 120 > > seconds. > > [ 2038.727766] Not tainted 3.14-0.bpo.1-amd64 #1 > > [ 2038.727846] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 2038.727966] nfsd D ffff88003fc14340 0 2747 2 > > 0x00000000 > > [ 2038.727970] ffff88003d1d0450 0000000000000046 0000000000000001 > > ffff88002f2502d0 > > [ 2038.727973] 0000000000014340 ffff88003c7d7fd8 0000000000014340 > > ffff88003d1d0450 > > [ 2038.727976] 0000000000000001 ffff88003fc14bd0 ffff88003d1d0450 > > ffffffff8112d400 > > [ 2038.727979] Call Trace: > > [ 2038.727984] [<ffffffff8112d400>] ? __lock_page+0x70/0x70 > > [ 2038.727989] [<ffffffff814efb97>] ? io_schedule+0x87/0xd0 > > [ 2038.727992] [<ffffffff8112d409>] ? sleep_on_page+0x9/0x10 > > [ 2038.727996] [<ffffffff814f0182>] ? __wait_on_bit+0x52/0x80 > > [ 2038.728000] [<ffffffff8112e0f7>] ? find_get_pages_tag+0xc7/0x180 > > [ 2038.728004] [<ffffffff8112d543>] ? wait_on_page_bit+0x73/0x80 > > [ 2038.728008] [<ffffffff810a6830>] ? > > wake_atomic_t_function+0x30/0x30 > > [ 2038.728011] [<ffffffff8112d624>] ? > > filemap_fdatawait_range+0xd4/0x150 > > [ 2038.728015] [<ffffffff8112ec18>] ? > > filemap_write_and_wait_range+0x48/0x90 > > [ 2038.728024] [<ffffffffa049b6e3>] ? xfs_setattr_size+0x2d3/0x3b0 > > [xfs] > > [ 2038.728028] [<ffffffff814f1a19>] ? down_write+0x9/0x30 > > [ 2038.728038] [<ffffffffa04d7608>] ? xfs_ilock+0xd8/0x160 [xfs] > > [ 2038.728047] [<ffffffffa049b818>] ? xfs_vn_setattr+0x58/0x70 > > [xfs] > > [ 2038.728051] [<ffffffff811b309b>] ? notify_change+0x1eb/0x3d0 > > [ 2038.728057] [<ffffffffa03ed5fc>] ? nfsd_setattr+0x24c/0x4b0 > > [nfsd] > > [ 2038.728064] [<ffffffffa03f5653>] ? nfsd3_proc_setattr+0x73/0xc0 > > [nfsd] > > [ 2038.728068] [<ffffffffa03e6d74>] ? nfsd_dispatch+0xe4/0x230 > > [nfsd] > > [ 2038.728075] [<ffffffffa034ad64>] ? > > svc_process_common+0x354/0x690 > > [sunrpc] > > [ 2038.728080] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > > [ 2038.728086] [<ffffffffa034b3fb>] ? svc_process+0x10b/0x160 > > [sunrpc] > > [ 2038.728091] [<ffffffffa03e66d7>] ? nfsd+0xb7/0x130 [nfsd] > > [ 2038.728095] [<ffffffffa03e6620>] ? nfsd_destroy+0x70/0x70 [nfsd] > > [ 2038.728098] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > > [ 2038.728102] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2038.728105] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > > [ 2038.728109] [<ffffffff81086cb0>] ? > > flush_kthread_worker+0xa0/0xa0 > > [ 2676.876333] [sched_delayed] sched: RT throttling activated > Well, these don't point at rbd at all. Are you seeing *any* progress > when this happens? Could it be that things just get very slow and > don't actually hang? > Can you try watching sysfs osdc file for a while to see if requests > are > going through or not? (/sys/kernel/debug/ceph/<fsid>.<id>/osdc) > Thanks, > Ilya > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140924/1dfdf62a/attachment.htm>
 |