I just added osd_heartbeat_grace to the [mon] section of ceph.conf, restarted ceph-mon, and now the monitor is reporting a 35 second osd_heartbeat_grace: [root at ceph-mon01 ceph]# ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon01.asok config show | grep osd_heartbeat_grace "osd_heartbeat_grace": "35", [root at ceph-mon01 ceph]# -----Original Message----- From: Bruce McFarland Sent: Monday, August 25, 2014 10:46 AM To: 'Gregory Farnum' Cc: ceph-users at ceph.com Subject: RE: osd_heartbeat_grace set to 30 but osd's still fail for grace > 20 That's something that was been puzzling to me. The monitor ceph.conf is set to 35, but it's runtime config reports 20. I've restarted it after initial creation to try and get it to reload the ceph.conf settings, but it stays's at 20. [root at ceph-mon01 ceph]# ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon01.asok config show | grep osd_heartbeat_grace "osd_heartbeat_grace": "20", [root at ceph-mon01 ceph]# [root at ceph-mon01 ceph]# cat ceph.conf [global] auth_service_required = cephx filestore_xattr_use_omap = true auth_client_required = cephx auth_cluster_required = cephx mon_host = 209.243.160.84 mon_initial_members = ceph-mon01 fsid = 94bbb882-42e4-4a6c-bfda-125790616fcc osd_pool_default_pg_num = 4096 osd_pool_default_pgp_num = 4096 osd_pool_default_size = 3 # Write an object 3 times - number of replicas. osd_pool_default_min_size = 1 # Allow writing one copy in a degraded state. [mon] mon_osd_min_down_reporters = 2 [osd] debug_ms = 1 debug_osd = 20 public_network = 209.243.160.0/24 cluster_network = 10.10.50.0/24 osd_journal_size = 96000 osd_heartbeat_grace = 35 [osd.0] . . . -----Original Message----- From: Gregory Farnum [mailto:greg@xxxxxxxxxxx] Sent: Monday, August 25, 2014 10:39 AM To: Bruce McFarland Cc: ceph-users at ceph.com Subject: Re: osd_heartbeat_grace set to 30 but osd's still fail for grace > 20 On Sat, Aug 23, 2014 at 11:06 PM, Bruce McFarland <Bruce.McFarland at taec.toshiba.com> wrote: > I see osd?s being failed for heartbeat reporting > default > osd_heartbeat_grace of 20 but the run time config shows that the grace > is set to 30. Is there another variable for the osd or the mon I need > to set for the non default osd_heartbeat_grace of 30 to take effect? You need to also set the osd heartbeat grace on the monitors. If I were to guess, the OSDs are actually seeing each other as slow (after 30 seconds) and reporting it in, but the monitors have a grace of 20 seconds set so that's what they're using to generate output. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com
 |