What is the setting of sysctl kernel.pid_max?
You relly need to have this:
kernel.pid_max = 4194304
(I think it also sets this as well: kernel.threads-max = 4194304)
I think you are running out of processs IDs.
Jan
> On 08 Dec 2015, at 08:10, Benedikt Fraunhofer <fraunhofer@xxxxxxxxxx> wrote:
>
> Hello Cephers,
>
> lately, our ceph-cluster started to show some weird behavior:
>
> the osd boxes show a load of 5000-15000 before the osds get marked down.
> Usually the box is fully usable, even "apt-get dist-upgrade" runs smoothly,
> you can read and write to any disk, only things you can't do are strace the osd
> processes, sync or reboot.
>
> we only find some logs about the "xfsaild = XFS Access Item List Daemon"
> as hung_task warnings.
>
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016108]
> [<ffffffff81093790>] ? kthread_create_on_node+0x1c0/0x1c0
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016112] INFO: task
> xfsaild/dm-1:1445 blocked for more than 120 seconds.
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016329] Tainted:
> G C 3.19.0-39-generic #44~14.04.1-Ubuntu
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016558] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016802] xfsaild/dm-1
> D ffff8807faa03af8 0 1445 2 0x00000000
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016805]
> ffff8807faa03af8 ffff8808098989d0 0000000000013e80 ffff8807faa03fd8
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016808]
> 0000000000013e80 ffff88080bb775c0 ffff8808098989d0 ffff88011381b2a8
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016812]
> ffff8807faa03c50 7fffffffffffffff ffff8807faa03c48 ffff8808098989d0
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016815] Call Trace:
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016819]
> [<ffffffff817b2fd9>] schedule+0x29/0x70
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016823]
> [<ffffffff817b609c>] schedule_timeout+0x20c/0x280
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016826]
> [<ffffffff810a40a5>] ? sched_clock_cpu+0x85/0xc0
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016830]
> [<ffffffff810a0911>] ? try_to_wake_up+0x1f1/0x340
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016834]
> [<ffffffff817b3d04>] wait_for_completion+0xa4/0x170
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016836]
> [<ffffffff810a0ad0>] ? wake_up_state+0x20/0x20
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016840]
> [<ffffffff8108e86d>] flush_work+0xed/0x1c0
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016846]
> [<ffffffff8108acc0>] ? destroy_worker+0x90/0x90
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016870]
> [<ffffffffc06f556e>] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016873]
> [<ffffffff810daddb>] ? lock_timer_base.isra.36+0x2b/0x50
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016878]
> [<ffffffff810dbdcf>] ? try_to_del_timer_sync+0x4f/0x70
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016901]
> [<ffffffffc06f3980>] _xfs_log_force+0x60/0x270 [xfs]
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016904]
> [<ffffffff810daba0>] ? internal_add_timer+0x80/0x80
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016926]
> [<ffffffffc06f3bba>] xfs_log_force+0x2a/0x90 [xfs]
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016948]
> [<ffffffffc06fe340>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016970]
> [<ffffffffc06fe480>] xfsaild+0x140/0x5a0 [xfs]
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016992]
> [<ffffffffc06fe340>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.016996]
> [<ffffffff81093862>] kthread+0xd2/0xf0
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.017000]
> [<ffffffff81093790>] ? kthread_create_on_node+0x1c0/0x1c0
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.017005]
> [<ffffffff817b72d8>] ret_from_fork+0x58/0x90
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.017009]
> [<ffffffff81093790>] ? kthread_create_on_node+0x1c0/0x1c0
> Dec 7 15:36:32 ceph1-store204 kernel: [152066.017013] INFO: task
> xfsaild/dm-6:1616 blocked for more than 120 seconds.
>
> kswapd is also reported as hung, but we don't have swap on the osds.
>
> It looks like either all ceph-osd-threads are reporting in as willing to work,
> or it's the xfs-maintenance-process itself like described in [1,2]
>
> Usually if we aint fast enough setting no{out,scrub,deep-scrub} this
> has an avalanche
> effect where we usually end up ipmi-power-cycling half of the cluster
> because all the osd-nodes
> are busy doing nothing (according to iostat or top, exept the load).
>
> Is this a known bug for kernel 3.19.0-39 (ubuntu 14.04 with the vivid kernel)?
> Do the xfs-tweaks described here
> https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg25295.html
> (i know this is for a pull request modifying the write-paths)
> look decent or worth a try?
>
> Currently we're running with "back to defaults" and less load
> (desperate try with the filestore settings, didnt change anything)
> ceph.conf-osd section:
>
> [osd]
> filestore max sync interval = 15
> filestore min sync interval = 1
> osd max backfills = 1
> osd recovery op priority = 1
>
>
> as a baffled try to get it to survive more than a day at a stretch.
>
> Maybe kernel 4.2 is worth a try?
>
> Thx for any input
> Benedikt
>
>
> [1] https://www.reddit.com/r/linux/comments/18kvdb/xfsaild_is_creating_tons_of_system_threads_and/
> [2] http://serverfault.com/questions/497049/the-xfs-filesystem-is-broken-in-rhel-centos-6-x-what-can-i-do-about-it
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
