On Wed, 23 Jan 2013, Jens Kristian S?gaard wrote: > Hi Sage, > > > > I think the problem now is just that 'osd target transaction size' is > > I set it to 50, and that seems to have solved all my problems. > > After a day or so my cluster got to a HEALTH_OK state again. It has been > > running for a few days now without any crashes! > > Hmm, one of the OSDs crashed again, sadly. > > It logs: > > -2> 2013-01-23 18:01:23.563624 7f67524da700 1 heartbeat_map is_healthy > 'FileStore::op_tp thread 0x7f673affd700' had timed out after 60 > -1> 2013-01-23 18:01:23.563657 7f67524da700 1 heartbeat_map is_healthy > 'FileStore::op_tp thread 0x7f673affd700' had suicide timed out after 180 > 0> 2013-01-23 18:01:24.257996 7f67524da700 -1 common/HeartbeatMap.cc: In > function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const > char*, time_t)' thread 7f67524da700 time 2013-01-23 18:01:23.563677 > common/HeartbeatMap.cc: 78: FAILED assert(0 == "hit suicide timeout") > > > With this stack trace: > > ceph version 0.56.1-26-g3bd8f6b (3bd8f6b7235eb14cab778e3c6dcdc636aff4f539) > 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, > long)+0x2eb) [0x846ecb] > 2: (ceph::HeartbeatMap::is_healthy()+0x8e) [0x8476ae] > 3: (ceph::HeartbeatMap::check_touch_file()+0x28) [0x8478d8] > 4: (CephContextServiceThread::entry()+0x55) [0x8e0f45] > 5: /lib64/libpthread.so.0() [0x3cbc807d14] > 6: (clone()+0x6d) [0x3cbc0f167d] > > > I have saved the core file, if there's anything in there you need? > > Or do you think I just need to set the target transaction size even lower than > 50? Can you share the output from 'thread apply all bt' so we can see what it was doing? thanks! s -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
