Hi Wido, On Thu, 3 Jan 2013, Wido den Hollander wrote: > Hi, > > I updated my 10 node 40 OSD cluster from 0.48 to 0.56 yesterday evening and > found out this morning that I had 23 OSDs still up and in. > > Investigating some logs I found these messages: This sounds quite a bit #3714. You might give wip-3714 a try... sage > > ********************************************************************* > -8> 2013-01-02 21:13:40.528936 7f9eb177a700 1 heartbeat_map is_healthy > 'OSD::op_tp thread 0x7f9ea2f5d700' had timed out after 30 > -7> 2013-01-02 21:13:40.528985 7f9eb177a700 1 heartbeat_map is_healthy > 'OSD::op_tp thread 0x7f9ea375e700' had timed out after 30 > -6> 2013-01-02 21:13:41.311088 7f9eaff77700 10 monclient: > _send_mon_message to mon.pri at [2a00:f10:11b:cef0:230:48ff:fed3:b086]:6789/0 > -5> 2013-01-02 21:13:45.047220 7f9e92282700 0 -- > [2a00:f10:11b:cef0:225:90ff:fe32:cf64]:0/2882 >> > [2a00:f10:11b:cef0:225:90ff:fe33:49fe]:6805/2373 pipe(0x9d7ad80 sd=135 :0 > pgs=0 cs=0 l=1).fault > -4> 2013-01-02 21:13:45.049225 7f9e962c2700 0 -- > [2a00:f10:11b:cef0:225:90ff:fe32:cf64]:6801/2882 >> > [2a00:f10:11b:cef0:225:90ff:fe33:49fe]:6804/2373 pipe(0x9d99000 sd=104 :44363 > pgs=99 cs=1 l=0).fault with nothing to send, going to standby > -3> 2013-01-02 21:13:45.529075 7f9eb177a700 1 heartbeat_map is_healthy > 'OSD::op_tp thread 0x7f9ea2f5d700' had timed out after 30 > -2> 2013-01-02 21:13:45.529115 7f9eb177a700 1 heartbeat_map is_healthy > 'OSD::op_tp thread 0x7f9ea2f5d700' had suicide timed out after 300 > -1> 2013-01-02 21:13:45.531952 7f9eb177a700 -1 common/HeartbeatMap.cc: In > function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const > char*, time_t)' thread 7f9eb177a700 time 2013-01-02 21:13:45.529176 > common/HeartbeatMap.cc: 78: FAILED assert(0 == "hit suicide timeout") > > ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8) > 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, > long)+0x107) [0x796877] > 2: (ceph::HeartbeatMap::is_healthy()+0x87) [0x797207] > 3: (ceph::HeartbeatMap::check_touch_file()+0x23) [0x797453] > 4: (CephContextServiceThread::entry()+0x55) [0x8338d5] > 5: (()+0x7e9a) [0x7f9eb4571e9a] > 6: (clone()+0x6d) [0x7f9eb2ff5cbd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > 0> 2013-01-02 21:13:46.314478 7f9eaff77700 10 monclient: > _send_mon_message to mon.pri at [2a00:f10:11b:cef0:230:48ff:fed3:b086]:6789/0 > ********************************************************************* > > Reading these messages I'm trying to figure out why those messages came along. > > Am I understanding this correctly that the heartbeat updates didn't come along > in time and the OSDs committed suicide? > > I read the code in common/HeartbeatMap.cc and it seems like that. > > During the restart of the cluster the Atom CPUs were very busy, so could it be > that the CPUs were just to busy and the OSDs weren't responding to heartbeats > in time? > > In total 16 of the 17 crashed OSDs are down with these log messages. > > I'm now starting the 16 crashed OSDs one by one and that seems to go just > fine. > > I've set "osd recovery max active = 1" to prevent overloading the CPUs to much > since I know Atoms are not that powerful. I'm just still trying to get it all > working on them :) > > Am I right this is probably a lack of CPU power during the heavy recovery > which causes them to not respond to heartbeat updates in time? > > Wido > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
