On Wed, Jul 1, 2015 at 8:38 AM, - - <francois.petit@xxxxxxxxxxxxxxxx> wrote: > Hi everybody, > > We have 3 monitors in our ceph cluster: 2 in one local site (2 data centers a > few km away from each other), and the 3rd one on a remote site, with a maximum > round-trip time (RTT) of 30ms between the local site and the remote site. All > OSDs run on the local site. The reason for the remote monitor is to keep the > cluster running if any DC fails. > > Is that a valid configuration ? What is the maximum RTT valid in such a Ceph > cluster ? > > > Here are some details about our running cluster: > Current monmap: > ----------- > epoch 4 > fsid ... > last_changed 2015-05-12 08:39:35.600843 > created 0.000000 > 0: <IP addr local0>:6789/0 mon.<local0> > 1: <IP addr local1>:6789/0 mon.<local1> > 2: <IP addr remote>:6789/0 mon.<remote> > ----------- > > In our running cluster, the mon logs show that the "leader" monitor is on the > local site, while the other two are "peons". > > Being curious, I increased runtime log-level debug settings for a few subsystems > (ms, mon, paxos...) to see if there was some kind of heartbeat between the > monitors. I noticed messages such as these ones... > ------ > 2015-07-01 07:01:05.840845 7fd569bbe700 1 -- <IP local1>:6789/0 --> mon.0 <IP > local0>:6789/0 -- mon_health( service 1 op tell e 0 r 0 ) v1 -- ?+0 0x3b9b200 > 2015-07-01 07:01:05.840871 7fd569bbe700 20 -- <IP local1>:6789/0 submit_message > mon_health( service 1 op tell e 0 r 0 ) v1 remote, <IP local0>:6789/0, have > pipe. > 2015-07-01 07:01:05.840885 7fd569bbe700 1 -- <IP local1>:6789/0 --> mon.2 <IP > remote>:6789/0 -- mon_health( service 1 op tell e 0 r 0 ) v1 -- ?+0 0x3b98a00 > 2015-07-01 07:01:05.840894 7fd569bbe700 20 -- <IP local1>:6789/0 submit_message > mon_health( service 1 op tell e 0 r 0 ) v1 remote, <IP remote>:6789/0, have > pipe. > ------ > ... but none which tells me what I want: the idea was to see if anybody could > complain about a high RTT, and to monitor that value. Any idea on how to do it ? I don't think there's anything that monitors RTT directly, but 30 ms shouldn't be a problem; that's an order of magnitude or more below all the various timeout thresholds. The clock skew detection might need to be loosened up but I'm not very familiar with how that bit works, and it's not quite as crucial anyway. :) -Greg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |