I know I'm not even close to this type of a problem yet with my small cluster (both test and production clusters) - but it would be great if something like that could appear in the cluster HEALTHWARN, if Ceph could determine the amount of used processes and compare them against the current limit then throw a health warning if it gets within say 10 or 15% of the max value. That would be a really quick indicator for anyone who frequently checks the health status (like through a web portal) as they may see it more quickly then during their regular log check interval. Just a thought.
-Tony
On Mon, Mar 9, 2015 at 2:01 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote:
On Mon, 9 Mar 2015, Karan Singh wrote:
> Thanks Guys kernel.pid_max=4194303 did the trick.
Great to hear! Sorry we missed that you only had it at 65536.
This is a really common problem that people hit when their clusters start
to grow. Is there somewhere in the docs we can put this to catch more
users? Or maybe a warning issued by the osds themselves or something if
they see limits that are low?
sage
> - Karan -
>
> On 09 Mar 2015, at 14:48, Christian Eichelmann
> <christian.eichelmann@xxxxxxxx> wrote:
>
> Hi Karan,
>
> as you are actually writing in your own book, the problem is the
> sysctl
> setting "kernel.pid_max". I've seen in your bug report that you were
> setting it to 65536, which is still to low for high density hardware.
>
> In our cluster, one OSD server has in an idle situation about 66.000
> Threads (60 OSDs per Server). The number of threads increases when you
> increase the number of placement groups in the cluster, which I think
> has triggered your problem.
>
> Set the "kernel.pid_max" setting to 4194303 (the maximum) like Azad
> Aliyar suggested, and the problem should be gone.
>
> Regards,
> Christian
>
> Am 09.03.2015 11:41, schrieb Karan Singh:
> Hello Community need help to fix a long going Ceph
> problem.
>
> Cluster is unhealthy , Multiple OSDs are DOWN. When i am
> trying to
> restart OSD?s i am getting this error
>
>
> /2015-03-09 12:22:16.312774 7f760dac9700 -1
> common/Thread.cc
> <http://Thread.cc>: In function 'void
> Thread::create(size_t)' thread
> 7f760dac9700 time 2015-03-09 12:22:16.311970/
> /common/Thread.cc <http://Thread.cc>: 129: FAILED
> assert(ret == 0)/
>
>
> *Environment *: 4 Nodes , OSD+Monitor , Firefly latest ,
> CentOS6.5
> , 3.17.2-1.el6.elrepo.x86_64
>
> Tried upgrading from 0.80.7 to 0.80.8 but no Luck
>
> Tried centOS stock kernel 2.6.32 but no Luck
>
> Memory is not a problem more then 150+GB is free
>
>
> Did any one every faced this problem ??
>
> *Cluster status *
> *
> *
> / cluster 2bd3283d-67ef-4316-8b7e-d8f4747eae33/
> / health HEALTH_WARN 7334 pgs degraded; 1185 pgs down;
> 1 pgs
> incomplete; 1735 pgs peering; 8938 pgs stale; 1/
> /736 pgs stuck inactive; 8938 pgs stuck stale; 10320 pgs
> stuck unclean;
> recovery 6061/31080 objects degraded (19/
> /.501%); 111/196 in osds are down; clock skew detected on
> mon.pouta-s02,
> mon.pouta-s03/
> / monmap e3: 3 mons at
> {pouta-s01=10.XXX.50.1:6789/0,pouta-s02=10.XXX.50.2:6789/0,pouta-s03=10.XXX
> .50.3:6789/
> //0}, election epoch 1312, quorum 0,1,2
> pouta-s01,pouta-s02,pouta-s03/
> / * osdmap e26633: 239 osds: 85 up, 196 in*/
> / pgmap v60389: 17408 pgs, 13 pools, 42345 MB data,
> 10360 objects/
> / 4699 GB used, 707 TB / 711 TB avail/
> / 6061/31080 objects degraded (19.501%)/
> / 14 down+remapped+peering/
> / 39 active/
> / 3289 active+clean/
> / 547 peering/
> / 663 stale+down+peering/
> / 705 stale+active+remapped/
> / 1 active+degraded+remapped/
> / 1 stale+down+incomplete/
> / 484 down+peering/
> / 455 active+remapped/
> / 3696 stale+active+degraded/
> / 4 remapped+peering/
> / 23 stale+down+remapped+peering/
> / 51 stale+active/
> / 3637 active+degraded/
> / 3799 stale+active+clean/
>
> *OSD : Logs *
>
> /2015-03-09 12:22:16.312774 7f760dac9700 -1
> common/Thread.cc
> <http://Thread.cc>: In function 'void
> Thread::create(size_t)' thread
> 7f760dac9700 time 2015-03-09 12:22:16.311970/
> /common/Thread.cc <http://Thread.cc>: 129: FAILED
> assert(ret == 0)/
> /
> /
> / ceph version 0.80.8
> (69eaad7f8308f21573c604f121956e64679a52a7)/
> / 1: (Thread::create(unsigned long)+0x8a) [0xaf41da]/
> / 2: (SimpleMessenger::add_accept_pipe(int)+0x6a)
> [0xae84fa]/
> / 3: (Accepter::entry()+0x265) [0xb5c635]/
> / 4: /lib64/libpthread.so.0() [0x3c8a6079d1]/
> / 5: (clone()+0x6d) [0x3c8a2e89dd]/
> / NOTE: a copy of the executable, or `objdump -rdS
> <executable>` is
> needed to interpret this./
>
>
> *More information at Ceph Tracker Issue :
> *http://tracker.ceph.com/issues/10988#change-49018
>
>
> ****************************************************************
> Karan Singh
> Systems Specialist , Storage Platforms
> CSC - IT Center for Science,
> Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
> mobile: +358 503 812758
> tel. +358 9 4572001
> fax +358 9 4572302
> http://www.csc.fi/
> ****************************************************************
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Christian Eichelmann
> Systemadministrator
>
> 1&1 Internet AG - IT Operations Mail & Media Advertising & Targeting
> Brauerstraße 48 · DE-76135 Karlsruhe
> Telefon: +49 721 91374-8026
> christian.eichelmann@xxxxxxxx
>
> Amtsgericht Montabaur / HRB 6484
> Vorstände: Henning Ahlert, Ralph Dommermuth, Matthias Ehrlich, Robert
> Hoffmann, Markus Huhn, Hans-Henning Kettler, Dr. Oliver Mauss, Jan
> Oetjen
> Aufsichtsratsvorsitzender: Michael Scheeren
>
>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
