So basically the issue - http://tracker.ceph.com/issues/4698 osd suicide timeout On Mon, Feb 22, 2016 at 7:06 PM, M Ranga Swami Reddy <swamireddy@xxxxxxxxx> wrote: > Hello, > I have reduced the scan_min and scan_max as below. After the below > change, during the scrubbing, got the op_tp_thread time out after 15. > After some time, OSDs crashed also... Any suggestions will be > helpful... Thanking you. > == > -osd_backfill_scan_min = 64 > -osd_backfill_scan_max = 512 > +osd_backfill_scan_min = 16 > +osd_backfill_scan_max = 32 > osd_op_thread_timeout = 15 > osd_recovery_thread_timeout = 30 > == > > emerg: 2016-02-22 02:48:24.301522 7f8ee3096700 1 heartbeat_map > is_healthy 'OSD::op_tp thread 0x7f8edf08e700' had timed out after 15 > > 2016-02-22T02:54:20.325657+00:00 emerg: --- begin dump of recent events --- > > … > > 2016-02-22T02:54:20.435546+00:00 emerg: 2016-02-22 02:54:20.315656 > 7f31a2166700 1 heartbeat_map reset_timeout 'OSD::op_tp thread > 0x7f31a2166700' had suicide timed out after 150 > > > > 2016-02-22T02:54:20.456737+00:00 emerg: 2016-02-22 02:54:20.237829 > 7fa09e05f700 -1 common/HeartbeatMap.cc: In function 'bool > ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, > time_t)' thread 7fa09e05f700 time 2016-02-22 02:54:20.187089 > > common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") > > > > ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) > > 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, > long)+0x107) [0x9c5757] > > 2: (ceph::HeartbeatMap::is_healthy()+0xa7) [0x9c6107] > > 3: (OSD::handle_osd_ping(MOSDPing*)+0x81b) [0x647a5b] > > 4: (OSD::heartbeat_dispatch(Message*)+0x51b) [0x648d0b] > > 5: (DispatchQueue::entry()+0x549) [0xb27269] > > 6: (DispatchQueue::DispatchThread::entry()+0xd) [0xa618ad] > > 7: (()+0x7e9a) [0x7fa0b4e77e9a] > > 8: (clone()+0x6d) [0x7fa0b3d708bd] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |