> Op 15 september 2016 om 13:27 schreef Kostis Fardelas <dante1234@xxxxxxxxx>: > > > Hello cephers, > being in a degraded cluster state with 6/162 OSDs down ((Hammer > 0.94.7, 162 OSDs, 27 "fat" nodes, 1000s of clients) ) like the below > ceph cluster log indicates: > > 2016-09-12 06:26:08.443152 mon.0 62.217.119.14:6789/0 217309 : cluster > [INF] pgmap v106027148: 28672 pgs: 2 down+remapped+peering, 25904 > active+clean, 23 stale+down+peering, 1 active+recovery_wait+degraded, > 1 active+recovery_wait+undersized+degraded, 170 down+peering, 1 > active+clean+scrubbing, 8 > active+undersized+degraded+remapped+wait_backfill, 27 > stale+active+undersized+degraded, 3 active+remapped+wait_backfill, > 2531 active+undersized+degraded, 1 > active+recovering+undersized+degraded+remapped; 95835 GB data, 186 TB > used, 94341 GB / 278 TB avail; 11230 B/s rd, 164 kB/s wr, 42 op/s; > 3148226/69530815 objects degraded (4.528%); 59272/69530815 objects > misplaced (0.085%); 1/34756893 unfound (0.000%) > > we experienced extensive thread leaks on the remaining up+in OSDs, > which lead to even more random crashes with Thread::create asserts: > > 2016-09-10 09:08:40.211713 7f8576bd6700 -1 common/Thread.cc: In > function 'void Thread::create(size_t)' thread 7f8576bd6700 time > 2016-09-10 09:08:40.199211 > common/Thread.cc: 131: FAILED assert(ret == 0) > > Thread count under normal operations are ~6500 on all nodes, but in > this degraded state we reached even ~35000. > > Is this expected behaviour when you have down+peering OSDs? > Is it possible to mitigate this problem using ceph configuration or > our only resort is kernel pid_max bump? > You should bump that setting. The default 32k is way to low during recovery. Set it to at least 512k or so. Wido > Regards, > Kostis > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |