Hello cephers, being in a degraded cluster state with 6/162 OSDs down ((Hammer 0.94.7, 162 OSDs, 27 "fat" nodes, 1000s of clients) ) like the below ceph cluster log indicates: 2016-09-12 06:26:08.443152 mon.0 62.217.119.14:6789/0 217309 : cluster [INF] pgmap v106027148: 28672 pgs: 2 down+remapped+peering, 25904 active+clean, 23 stale+down+peering, 1 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded, 170 down+peering, 1 active+clean+scrubbing, 8 active+undersized+degraded+remapped+wait_backfill, 27 stale+active+undersized+degraded, 3 active+remapped+wait_backfill, 2531 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped; 95835 GB data, 186 TB used, 94341 GB / 278 TB avail; 11230 B/s rd, 164 kB/s wr, 42 op/s; 3148226/69530815 objects degraded (4.528%); 59272/69530815 objects misplaced (0.085%); 1/34756893 unfound (0.000%) we experienced extensive thread leaks on the remaining up+in OSDs, which lead to even more random crashes with Thread::create asserts: 2016-09-10 09:08:40.211713 7f8576bd6700 -1 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f8576bd6700 time 2016-09-10 09:08:40.199211 common/Thread.cc: 131: FAILED assert(ret == 0) Thread count under normal operations are ~6500 on all nodes, but in this degraded state we reached even ~35000. Is this expected behaviour when you have down+peering OSDs? Is it possible to mitigate this problem using ceph configuration or our only resort is kernel pid_max bump? Regards, Kostis _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |