Hi! I got Ceph Hammer 0.94.3 with 72 OSD and 6 nodes. My problem is that i'm constantly getting my /var/log/messages full of such messages: Sep 3 11:16:31 slpeah001 ceph-osd: 2015-09-03 11:16:31.393234 7f5a6bfd3700 -1 osd.34 2991 heartbeat_check: no reply from osd.68 since back 2015-09-03 11:16:25.491465 front 2015-09-03 11:16:25.491465 (cutoff 2015-09-03 11:16:26.392797) Sep 3 11:16:31 slpeah001 ceph-osd: 2015-09-03 11:16:31.393239 7f5a6bfd3700 -1 osd.34 2991 heartbeat_check: no reply from osd.69 since back 2015-09-03 11:16:25.491465 front 2015-09-03 11:16:25.491465 (cutoff 2015-09-03 11:16:26.392797) Sep 3 11:16:31 slpeah001 ceph-osd: 2015-09-03 11:16:31.393244 7f5a6bfd3700 -1 osd.34 2991 heartbeat_check: no reply from osd.70 since back 2015-09-03 11:16:25.491465 front 2015-09-03 11:16:25.491465 (cutoff 2015-09-03 11:16:26.392797) Sep 3 11:16:31 slpeah001 ceph-osd: 2015-09-03 11:16:31.393249 7f5a6bfd3700 -1 osd.34 2991 heartbeat_check: no reply from osd.71 since back 2015-09-03 11:16:25.491465 front 2015-09-03 11:16:25.491465 (cutoff 2015-09-03 11:16:26.392797) This messages keep spawning until my /var/log/ filesystem is depleted. What do these messages mean? Is this a sign of a network issue? How can I get rid of them? Because the grow rate is as high as several Gb a day. I tried to tune my configuration WRT logging: [global] log_to_syslog = false log_to_stderr = false mon_cluster_log_file = /dev/null mon_cluster_log_to_syslog = false mon_cluster_log_file_level = warn mon_cluster_log_to_syslog_level = warn clog_to_syslog_level = warn debug lockdep = 0/0 debug context = 0/0 debug buffer = 0/0 debug timer = 0/0 debug osd = 0/0 debug optracker = 0/0 debug objclass = 0/0 debug filestore = 0/0 debug ms = 0/0 debug monc = 0/0 debug throttle = 0/0 debug_finisher = 0/0 debug_heartbeatmap = 0/0 debug_rgw = 0/0 debug_civetweb = 0/0 debug_javaclient = 0/0 But that didn't help. What else can I do? Regards, Vasily. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |