Hi all. Looking at the archives, this seems a development mailing list, but since I don't know better place where put this question, I'll let you know the problem. I have an old Debian 6.0 machine that served as a NFS server for a HPC cluster. Users started to complain that the machines that accessed the NFS got really slow (/home, for instance, is NFS-mounted). After some investigation I realized that all nfsd process are rapidly going to 'D' state and rarely coming back. Some investigation led me to the hypothesis that it could be a bug in Linux kernel (see [1]). So, I decided to update the old kernel (2.6.32-3) to a newer one (3.2.60). It didn't solve. Some changes of the nfs mount options, like using the "soft" option, or tweaking wsize and rsize parameters didn't solve as well. After that, I tried to increase the number of nfsd's from 16 to 64. No way. All of them stuck to 'D' again is not a throttle problem in disk. I/O rates don't go beyond 20 MB/s, according to iostat. This is an architecture where 22 computers mount the same directory (/home) and share information there. Although a distributed filesystem is recommended for this case (HPC), according to our experience NFS suffices for this size of cluster. For some information, I give you here: 1. output of nfsstat. 2. dmesg with some backtraces. Thank you! 1. output of nfsstat Server rpc stats: calls badcalls badauth badclnt xdrcall 90589 1 1 0 0 Server nfs v3: null getattr setattr lookup access readlink 57 0% 16334 18% 10186 11% 1513 1% 5442 6% 0 0% read write create mkdir symlink mknod 6 0% 42268 46% 268 0% 2 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 259 0% 0 0% 279 0% 0 0% 0 0% 70 0% fsstat fsinfo pathconf commit 29 0% 26 0% 13 0% 13697 15% Client rpc stats: calls retrans authrefrsh 0 0 0 2. dmesg with some backtraces (you realize that there are some other processes in 'D', like a sync command I executed and didn't return. Disk problem? kjournald D ffff880610e48dc0 0 790 2 0x00000000 ffff880610e48dc0 0000000000000046 ffff880600000000 ffff880613d01b80 0000000000013140 ffff88061257ffd8 ffff88061257ffd8 0000000000013140 ffff880610e48dc0 0000000000013140 0000000000013140 ffff88061257e010 Call Trace: [<ffffffff8101485d>] ? read_tsc+0x5/0x16 [<ffffffff8111ee09>] ? unmap_underlying_metadata+0x4b/0x4b [<ffffffff8132b4cc>] ? io_schedule+0x84/0xc3 [<ffffffff8111ee12>] ? sleep_on_buffer+0x9/0xd [<ffffffff8132b8c7>] ? __wait_on_bit+0x3e/0x6f [<ffffffff8132b966>] ? out_of_line_wait_on_bit+0x6e/0x77 [<ffffffff8111ee09>] ? unmap_underlying_metadata+0x4b/0x4b [<ffffffff81062fe7>] ? autoremove_wake_function+0x2a/0x2a [<ffffffffa02fbb3e>] ? wait_on_buffer+0xe/0x28 [jbd] [<ffffffffa02fc22f>] ? journal_commit_transaction+0x597/0xec8 [jbd] [<ffffffff81062fbd>] ? wake_up_bit+0x20/0x20 [<ffffffff81054bcf>] ? lock_timer_base+0x25/0x49 [<ffffffffa03001eb>] ? kjournald+0xde/0x220 [jbd] [<ffffffff81062fbd>] ? wake_up_bit+0x20/0x20 [<ffffffffa030010d>] ? commit_timeout+0xa/0xa [jbd] [<ffffffffa030010d>] ? commit_timeout+0xa/0xa [jbd] [<ffffffff81062b71>] ? kthread+0x7a/0x82 [<ffffffff8132f234>] ? kernel_thread_helper+0x4/0x10 [<ffffffff81062af7>] ? kthread_worker_fn+0x147/0x147 [<ffffffff8132f230>] ? gs_change+0x13/0x13 nfsd D ffff880610f5d280 0 1336 2 0x00000000 ffff880610f5d280 0000000000000046 0000000000000000 ffff880613d06e00 0000000000013140 ffff880607aa7fd8 ffff880607aa7fd8 0000000000013140 ffff880610f5d280 0000000000013140 0000000000013140 ffff880607aa6010 Call Trace: [<ffffffff8101485d>] ? read_tsc+0x5/0x16 [<ffffffff810b9fdd>] ? __lock_page+0x63/0x63 [<ffffffff8132b4cc>] ? io_schedule+0x84/0xc3 [<ffffffff8119b6f8>] ? radix_tree_gang_lookup_slot+0x85/0xa9 [<ffffffff810b9fe6>] ? sleep_on_page+0x9/0xd [<ffffffff8132b8c7>] ? __wait_on_bit+0x3e/0x6f [<ffffffff810ba194>] ? wait_on_page_bit+0x6a/0x70 [<ffffffff81062fe7>] ? autoremove_wake_function+0x2a/0x2a [<ffffffff810c44eb>] ? truncate_inode_pages_range+0x2c6/0x317 [<ffffffff810c4581>] ? truncate_pagecache+0x36/0x4c [<ffffffffa036e199>] ? ext3_setattr+0x271/0x2c6 [ext3] [<ffffffff8110e46d>] ? notify_change+0x1c5/0x296 [<ffffffffa042ec71>] ? nfsd_setattr+0x296/0x2e0 [nfsd] [<ffffffffa043417c>] ? nfsd3_proc_setattr+0x64/0x6e [nfsd] [<ffffffffa042a2fe>] ? nfsd_dispatch+0xd9/0x1bb [nfsd] [<ffffffffa03a6ea2>] ? svc_process+0x422/0x626 [sunrpc] [<ffffffffa042a790>] ? nfsd_svc+0x167/0x167 [nfsd] [<ffffffffa042a873>] ? nfsd+0xe3/0x127 [nfsd] [<ffffffff81062b71>] ? kthread+0x7a/0x82 [<ffffffff8132f234>] ? kernel_thread_helper+0x4/0x10 [<ffffffff81062af7>] ? kthread_worker_fn+0x147/0x147 [<ffffffff8132f230>] ? gs_change+0x13/0x13 nfsd D ffff880610f5e720 0 1340 2 0x00000000 ffff880610f5e720 0000000000000046 0000000000000000 ffffffff8160d020 0000000000013140 ffff8806076b7fd8 ffff8806076b7fd8 0000000000013140 ffff880610f5e720 0000000000013140 0000000000013140 ffff8806076b6010 Call Trace: [<ffffffff810b9fdd>] ? __lock_page+0x63/0x63 [<ffffffff8132b4cc>] ? io_schedule+0x84/0xc3 [<ffffffff810b9fe6>] ? sleep_on_page+0x9/0xd [<ffffffff8132b8c7>] ? __wait_on_bit+0x3e/0x6f [<ffffffffa02fb383>] ? do_get_write_access+0x36d/0x3b4 [jbd] [<ffffffff810ba194>] ? wait_on_page_bit+0x6a/0x70 [<ffffffff81062fe7>] ? autoremove_wake_function+0x2a/0x2a [<ffffffff810bb4e6>] ? grab_cache_page_write_begin+0x9f/0xb1 [<ffffffffa036e928>] ? ext3_write_begin+0xae/0x20e [ext3] [<ffffffff810b96cf>] ? generic_file_buffered_write+0x121/0x28e [<ffffffff810ba975>] ? __generic_file_aio_write+0x243/0x277 [<ffffffff810ba9a9>] ? __generic_file_aio_write+0x277/0x277 [<ffffffff810baa08>] ? generic_file_aio_write+0x5f/0xb3 [<ffffffff810f9d55>] ? do_sync_readv_writev+0xa3/0xde [<ffffffffa042caa6>] ? fh_compose+0x33f/0x33f [nfsd] [<ffffffffa00735e5>] ? exportfs_decode_fh+0xc7/0x226 [exportfs] [<ffffffffa03adfdb>] ? seconds_since_boot+0x11/0x1a [sunrpc] [<ffffffff810f9a3b>] ? copy_from_user+0x18/0x30 [<ffffffff81165106>] ? security_file_permission+0x18/0x2d [<ffffffff810fa3b1>] ? do_readv_writev+0xaf/0x125 [<ffffffff8113c9a4>] ? dqget+0x2c0/0x31b [<ffffffff8113cf00>] ? dquot_initialize+0x6/0xd [<ffffffff8113ced3>] ? __dquot_initialize+0x145/0x16c [<ffffffff8113beb7>] ? dqput+0x3b/0x1b7 [<ffffffffa042de88>] ? nfsd_vfs_write+0xc9/0x2b5 [nfsd] [<ffffffffa042e38b>] ? nfsd_open+0xfd/0x14d [nfsd] [<ffffffffa042e6f3>] ? nfsd_write+0xbb/0xdc [nfsd] [<ffffffffa0433e72>] ? nfsd3_proc_write+0xcf/0xea [nfsd] [<ffffffffa042a2fe>] ? nfsd_dispatch+0xd9/0x1bb [nfsd] [<ffffffffa03a6ea2>] ? svc_process+0x422/0x626 [sunrpc] [<ffffffffa042a790>] ? nfsd_svc+0x167/0x167 [nfsd] [<ffffffffa042a873>] ? nfsd+0xe3/0x127 [nfsd] [<ffffffff81062b71>] ? kthread+0x7a/0x82 [<ffffffff8132f234>] ? kernel_thread_helper+0x4/0x10 [<ffffffff81062af7>] ? kthread_worker_fn+0x147/0x147 [<ffffffff8132f230>] ? gs_change+0x13/0x13 nfsd D ffff880610e4d280 0 1346 2 0x00000000 ffff880610e4d280 0000000000000046 0000000000000000 ffff880613d06e00 0000000000013140 ffff880606ce9fd8 ffff880606ce9fd8 0000000000013140 ffff880610e4d280 0000000000013140 0000000000013140 ffff880606ce8010 Call Trace: [<ffffffff810b9fdd>] ? __lock_page+0x63/0x63 [<ffffffff8132b4cc>] ? io_schedule+0x84/0xc3 [<ffffffff810b9fe6>] ? sleep_on_page+0x9/0xd [<ffffffff8132b8c7>] ? __wait_on_bit+0x3e/0x6f [<ffffffffa02fb383>] ? do_get_write_access+0x36d/0x3b4 [jbd] [<ffffffff810ba194>] ? wait_on_page_bit+0x6a/0x70 [<ffffffff81062fe7>] ? autoremove_wake_function+0x2a/0x2a [<ffffffff810bb4e6>] ? grab_cache_page_write_begin+0x9f/0xb1 [<ffffffffa036e928>] ? ext3_write_begin+0xae/0x20e [ext3] [<ffffffff810b96cf>] ? generic_file_buffered_write+0x121/0x28e [<ffffffff810ba975>] ? __generic_file_aio_write+0x243/0x277 [<ffffffff810ba9a9>] ? __generic_file_aio_write+0x277/0x277 [<ffffffff810baa08>] ? generic_file_aio_write+0x5f/0xb3 [<ffffffff810f9d55>] ? do_sync_readv_writev+0xa3/0xde [<ffffffffa042caa6>] ? fh_compose+0x33f/0x33f [nfsd] [<ffffffffa00735e5>] ? exportfs_decode_fh+0xc7/0x226 [exportfs] [<ffffffffa03adfdb>] ? seconds_since_boot+0x11/0x1a [sunrpc] [<ffffffff810f9a3b>] ? copy_from_user+0x18/0x30 [<ffffffff81165106>] ? security_file_permission+0x18/0x2d [<ffffffff810fa3b1>] ? do_readv_writev+0xaf/0x125 [<ffffffff8113c9a4>] ? dqget+0x2c0/0x31b [<ffffffff8113cf00>] ? dquot_initialize+0x6/0xd [<ffffffff8113ced3>] ? __dquot_initialize+0x145/0x16c [<ffffffff8113beb7>] ? dqput+0x3b/0x1b7 [<ffffffffa042de88>] ? nfsd_vfs_write+0xc9/0x2b5 [nfsd] [<ffffffffa042e38b>] ? nfsd_open+0xfd/0x14d [nfsd] [<ffffffffa042e6f3>] ? nfsd_write+0xbb/0xdc [nfsd] [<ffffffffa0433e72>] ? nfsd3_proc_write+0xcf/0xea [nfsd] [<ffffffffa042a2fe>] ? nfsd_dispatch+0xd9/0x1bb [nfsd] [<ffffffffa03a6ea2>] ? svc_process+0x422/0x626 [sunrpc] [<ffffffffa042a790>] ? nfsd_svc+0x167/0x167 [nfsd] [<ffffffffa042a873>] ? nfsd+0xe3/0x127 [nfsd] [<ffffffff81062b71>] ? kthread+0x7a/0x82 [<ffffffff8132f234>] ? kernel_thread_helper+0x4/0x10 [<ffffffff81062af7>] ? kthread_worker_fn+0x147/0x147 [<ffffffff8132f230>] ? gs_change+0x13/0x13 sync D ffff880610a11b80 0 6556 6486 0x00000004 ffff880610a11b80 0000000000000086 0000000000000000 ffff880613d00000 0000000000013140 ffff8805f8515fd8 ffff8805f8515fd8 0000000000013140 ffff880610a11b80 0000000000013140 0000000000013140 ffff8805f8514010 Call Trace: [<ffffffff8132b693>] ? schedule_timeout+0x2d/0xd7 [<ffffffff810f126b>] ? virt_to_head_page+0x9/0x2c [<ffffffff810f3712>] ? kmem_cache_free+0x12/0x9e [<ffffffff8132b2d9>] ? wait_for_common+0xd1/0x14e [<ffffffff81045664>] ? try_to_wake_up+0x191/0x191 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111923d>] ? writeback_inodes_sb_nr+0x7b/0x81 [<ffffffff8111c8ef>] ? __sync_filesystem+0x53/0x79 [<ffffffff810fbf1e>] ? iterate_supers+0x5e/0xab [<ffffffff8111c956>] ? sys_sync+0x2d/0x58 [<ffffffff8132d0d2>] ? system_call_fastpath+0x16/0x1b sync D ffff880610a11b80 0 6556 6486 0x00000004 ffff880610a11b80 0000000000000086 0000000000000000 ffff880613d00000 0000000000013140 ffff8805f8515fd8 ffff8805f8515fd8 0000000000013140 ffff880610a11b80 0000000000013140 0000000000013140 ffff8805f8514010 Call Trace: [<ffffffff8132b693>] ? schedule_timeout+0x2d/0xd7 [<ffffffff810f126b>] ? virt_to_head_page+0x9/0x2c [<ffffffff810f3712>] ? kmem_cache_free+0x12/0x9e [<ffffffff8132b2d9>] ? wait_for_common+0xd1/0x14e [<ffffffff81045664>] ? try_to_wake_up+0x191/0x191 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111923d>] ? writeback_inodes_sb_nr+0x7b/0x81 [<ffffffff8111c8ef>] ? __sync_filesystem+0x53/0x79 [<ffffffff810fbf1e>] ? iterate_supers+0x5e/0xab [<ffffffff8111c956>] ? sys_sync+0x2d/0x58 [<ffffffff8132d0d2>] ? system_call_fastpath+0x16/0x1b sync D ffff880610a11b80 0 6556 6486 0x00000004 ffff880610a11b80 0000000000000086 0000000000000000 ffff880613d00000 0000000000013140 ffff8805f8515fd8 ffff8805f8515fd8 0000000000013140 ffff880610a11b80 0000000000013140 0000000000013140 ffff8805f8514010 Call Trace: [<ffffffff8132b693>] ? schedule_timeout+0x2d/0xd7 [<ffffffff810f126b>] ? virt_to_head_page+0x9/0x2c [<ffffffff810f3712>] ? kmem_cache_free+0x12/0x9e [<ffffffff8132b2d9>] ? wait_for_common+0xd1/0x14e [<ffffffff81045664>] ? try_to_wake_up+0x191/0x191 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111923d>] ? writeback_inodes_sb_nr+0x7b/0x81 [<ffffffff8111c8ef>] ? __sync_filesystem+0x53/0x79 [<ffffffff810fbf1e>] ? iterate_supers+0x5e/0xab [<ffffffff8111c956>] ? sys_sync+0x2d/0x58 [<ffffffff8132d0d2>] ? system_call_fastpath+0x16/0x1b kjournald D ffff880610e48dc0 0 790 2 0x00000000 ffff880610e48dc0 0000000000000046 ffff880600000000 ffff880613d05280 0000000000013140 ffff88061257ffd8 ffff88061257ffd8 0000000000013140 ffff880610e48dc0 0000000000013140 0000000000013140 ffff88061257e010 Call Trace: [<ffffffff8101485d>] ? read_tsc+0x5/0x16 [<ffffffff8111ee09>] ? unmap_underlying_metadata+0x4b/0x4b [<ffffffff8132b4cc>] ? io_schedule+0x84/0xc3 [<ffffffff8111ee12>] ? sleep_on_buffer+0x9/0xd [<ffffffff8132b8c7>] ? __wait_on_bit+0x3e/0x6f [<ffffffff8132b966>] ? out_of_line_wait_on_bit+0x6e/0x77 [<ffffffff8111ee09>] ? unmap_underlying_metadata+0x4b/0x4b [<ffffffff81062fe7>] ? autoremove_wake_function+0x2a/0x2a [<ffffffffa02fbb3e>] ? wait_on_buffer+0xe/0x28 [jbd] [<ffffffffa02fc22f>] ? journal_commit_transaction+0x597/0xec8 [jbd] [<ffffffff81062fbd>] ? wake_up_bit+0x20/0x20 [<ffffffff81054bcf>] ? lock_timer_base+0x25/0x49 [<ffffffffa03001eb>] ? kjournald+0xde/0x220 [jbd] [<ffffffff81062fbd>] ? wake_up_bit+0x20/0x20 [<ffffffffa030010d>] ? commit_timeout+0xa/0xa [jbd] [<ffffffffa030010d>] ? commit_timeout+0xa/0xa [jbd] [<ffffffff81062b71>] ? kthread+0x7a/0x82 [<ffffffff8132f234>] ? kernel_thread_helper+0x4/0x10 [<ffffffff81062af7>] ? kthread_worker_fn+0x147/0x147 [<ffffffff8132f230>] ? gs_change+0x13/0x13 flush-8:16 D ffff8806106e2940 0 1009 2 0x00000000 ffff8806106e2940 0000000000000046 0000000000000000 ffff880613d01b80 0000000000013140 ffff88061196bfd8 ffff88061196bfd8 0000000000013140 ffff8806106e2940 0000000000013140 0000000000013140 ffff88061196a010 Call Trace: [<ffffffff8132b4cc>] ? io_schedule+0x84/0xc3 [<ffffffff81188985>] ? get_request_wait+0x104/0x193 [<ffffffff81062fbd>] ? wake_up_bit+0x20/0x20 [<ffffffff81188c78>] ? blk_queue_bio+0x17a/0x27a [<ffffffff8111ef7d>] ? T.1002+0x17/0x17 [<ffffffff81188300>] ? generic_make_request+0x8e/0xcd [<ffffffff81188418>] ? submit_bio+0xd9/0xf7 [<ffffffff8111ef7d>] ? T.1002+0x17/0x17 [<ffffffff8112153b>] ? bio_alloc_bioset+0x44/0xb5 [<ffffffff8111dbd0>] ? submit_bh+0xe5/0x105 [<ffffffff81120267>] ? __block_write_full_page+0x1dd/0x2b5 [<ffffffff810c0cc5>] ? __writepage+0xa/0x21 [<ffffffff810c1e27>] ? write_cache_pages+0x226/0x31e [<ffffffff810c0cbb>] ? set_page_dirty+0x61/0x61 [<ffffffff810c1f5d>] ? generic_writepages+0x3e/0x55 [<ffffffff81118a4f>] ? writeback_single_inode+0x17f/0x362 [<ffffffff81118efe>] ? writeback_sb_inodes+0x169/0x1ff [<ffffffff81119001>] ? __writeback_inodes_wb+0x6d/0xab [<ffffffff81119720>] ? wb_writeback+0x128/0x222 [<ffffffff810c149a>] ? bdi_dirty_limit+0x27/0x81 [<ffffffff8111989b>] ? wb_do_writeback+0x81/0x1dd [<ffffffff8105508b>] ? del_timer_sync+0x34/0x3e [<ffffffff81119aba>] ? bdi_writeback_thread+0xc3/0x201 [<ffffffff811199f7>] ? wb_do_writeback+0x1dd/0x1dd [<ffffffff811199f7>] ? wb_do_writeback+0x1dd/0x1dd [<ffffffff81062b71>] ? kthread+0x7a/0x82 [<ffffffff8132f234>] ? kernel_thread_helper+0x4/0x10 [<ffffffff81062af7>] ? kthread_worker_fn+0x147/0x147 [<ffffffff8132f230>] ? gs_change+0x13/0x13 sync D ffff880610a11b80 0 6556 6486 0x00000004 ffff880610a11b80 0000000000000086 0000000000000000 ffff880613d00000 0000000000013140 ffff8805f8515fd8 ffff8805f8515fd8 0000000000013140 ffff880610a11b80 0000000000013140 0000000000013140 ffff8805f8514010 Call Trace: [<ffffffff8132b693>] ? schedule_timeout+0x2d/0xd7 [<ffffffff810f126b>] ? virt_to_head_page+0x9/0x2c [<ffffffff810f3712>] ? kmem_cache_free+0x12/0x9e [<ffffffff8132b2d9>] ? wait_for_common+0xd1/0x14e [<ffffffff81045664>] ? try_to_wake_up+0x191/0x191 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111c915>] ? __sync_filesystem+0x79/0x79 [<ffffffff8111923d>] ? writeback_inodes_sb_nr+0x7b/0x81 [<ffffffff8111c8ef>] ? __sync_filesystem+0x53/0x79 [<ffffffff810fbf1e>] ? iterate_supers+0x5e/0xab [<ffffffff8111c956>] ? sys_sync+0x2d/0x58 [<ffffffff8132d0d2>] ? system_call_fastpath+0x16/0x1b nfsd: peername failed (err 107)! nfsd: last server has exited, flushing export cache svc: failed to register lockdv1 RPC service (errno 97). References: [1] http://administratosphere.wordpress.com/2011/05/13/linux-kernel-sync-bug/ -- Silas Silva -- Silas Silva -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
