Hi Gregory,
On 28/01/14 15:51, Gregory Farnum wrote:
>> I do note ntp doesn't seem to be doing its job, but that's a side issue.
> Actually, that could be it. If you take down one of the monitors and
> the other two have enough of a time gap that they won't talk to each
> other, your cluster won't be able to make any progress. The OSDs don't
> much care, but your monitor nodes need to have a well-synced clock.
I've done some more testing here. Had lots of fun and games learning
about the intricacies of running NTP peers (as opposed to servers) and
the like.
One thing I observe is that mon.0 seems to always dominate. I'll have
all three nodes up, kill mon.0 via the power switch, wait for the
clients to switch to another monitor, then power it back up.
Suddenly monitors 1 and 2 are the ones with the clocks askew, not
monitor 0 who's just loaded the time from the hardware clock and not yet
synchronised to NTP. After a while, it all sorts itself out (I guess
when NTP gets them within sync), but for a while, I get the HEALTH_WARN
state in ceph.
i.e. initially ceph health reports:
> HEALTH_OK
then I power off sn0, I get:
> HEALTH_WARN 759 pgs degraded; 387 pgs stuck unclean; recovery 102277/306831 objects degraded (33.333%); 2/6 in osds are
> down; 1 mons down, quorum 1,2 1,2
and ceph -w shows:
> 2014-01-29 12:39:12.127994 mon.1 [INF] pgmap v86707: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 17362 kB/s rd, 135 op/s; 102277/306831 objects degraded (33.333%)
> 2014-01-29 12:39:14.023174 mon.1 [INF] pgmap v86708: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 5340 kB/s rd, 41 op/s; 102277/306831 objects degraded (33.333%)
> 2014-01-29 12:39:16.032460 mon.1 [INF] pgmap v86709: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 4196 kB/s rd, 32 op/s; 102277/306831 objects degraded (33.333%)
I power up sn0 now, and after an OSD recovery:
> HEALTH_WARN clock skew detected on mon.1
and ceph -w shows:
> 2014-01-29 12:40:36.042565 mon.1 [INF] pgmap v86757: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 5124 kB/s rd, 40 op/s; 102277/306831 objects degraded (33.333%)
> 2014-01-29 12:40:37.047329 mon.1 [INF] pgmap v86758: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 15798 kB/s rd, 123 op/s; 102277/306831 objects degraded (33.333%)
> 2014-01-29 12:40:39.033544 mon.1 [INF] pgmap v86759: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 15920 kB/s rd, 124 op/s; 102277/306831 objects degraded (33.333%)
> 2014-01-29 12:40:40.556413 mon.0 [INF] mon.0 calling new monitor election
> 2014-01-29 12:40:40.562922 mon.0 [INF] mon.0 calling new monitor election
> 2014-01-29 12:40:40.563389 mon.0 [INF] mon.0@0 won leader election with quorum 0,1,2
> 2014-01-29 12:40:40.565821 mon.0 [WRN] mon.1 10.87.160.225:6789/0 clock skew 0.106539s > max 0.05s
> 2014-01-29 12:40:40.670626 mon.1 [INF] mon.1 calling new monitor election
> 2014-01-29 12:40:40.687451 mon.2 [INF] mon.2 calling new monitor election
> 2014-01-29 12:40:40.866972 mon.0 [INF] pgmap v86759: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 102277/306831 objects degraded (33.333%)
> 2014-01-29 12:40:40.867048 mon.0 [INF] mdsmap e1: 0/0/1 up
> 2014-01-29 12:40:40.867199 mon.0 [INF] osdmap e501: 6 osds: 4 up, 6 in
> 2014-01-29 12:40:40.867285 mon.0 [INF] monmap e1: 3 mons at {0=10.20.30.224:6789/0,1=10.20.30.225:6789/0,2=10.20.30.226:6789/0}
> 2014-01-29 12:40:40.962267 mon.0 [WRN] message from mon.2 was stamped 0.125828s in the future, clocks not synchronized
> 2014-01-29 12:40:41.003207 mon.0 [INF] osdmap e502: 6 osds: 4 up, 6 in
> 2014-01-29 12:40:41.006869 mon.0 [INF] pgmap v86760: 800 pgs: 41 active+remapped, 759 active+degraded; 399 GB data, 1199 GB used, 15559 GB / 16758 GB avail; 5168 kB/s rd, 40 op/s; 102277/306831 objects degraded (33.333%)
> 2014-01-29 12:40:41.971799 mon.0 [INF] osdmap e503: 6 osds: 4 up, 6 in
Presently, I've got three authorative NTP servers on the network:
- border router (Ubuntu Linux 12.04)
- interior router (Cisco IOS)
- file server (Ubuntu Linux 10.04)
I initially had the Ceph nodes synced to them, and to each other i.e. in
ntp.conf:
server ntp0.domain
server ntp1.domain
server ntp2.domain
peer a-mon-node
peer another-mon-node
Having discovered NTP doesn't like these synchronisation loops (where a
peer uses the same reference as another) I've since changed this to:
server ntp.domain # RR A record for ntp0, ntp1 and ntp2
server 0.au.pool.ntp.org
server 1.au.pool.ntp.org
server 2.au.pool.ntp.org
server 3.au.pool.ntp.org
peer a-mon-node
peer another-mon-node
That means they all pick a number of nodes at random from the public NTP
server pool. Not sure what tricks others have tried with regards to
NTP, but at least now the peers are marked as "candidates" and not
"rejected".
> And in your trace everything got wedged for so long the system just
> gave up; that's probably a result of the cluster having data it
> couldn't write to for too long. (Like I said before, you should make
> sure your CRUSH map and rules look right.)
At the moment my CRUSH map looks like this:
> root@sn2:~# ceph osd getcrushmap > crushmap.bin
> got crush map from osdmap epoch 489
> root@sn2:~# crushtool -d crushmap.bin > crushmap.txt
> root@sn2:~# cat crushmap.txt
> # begin crush map
>
> # devices
> device 0 osd.0
> device 1 osd.1
> device 2 osd.2
> device 3 osd.3
> device 4 osd.4
> device 5 osd.5
>
> # types
> type 0 osd
> type 1 host
> type 2 rack
> type 3 row
> type 4 room
> type 5 datacenter
> type 6 root
>
> # buckets
> host sn0 {
> id -2 # do not change unnecessarily
> # weight 5.460
> alg straw
> hash 0 # rjenkins1
> item osd.0 weight 2.730
> item osd.1 weight 2.730
> }
> host sn1 {
> id -3 # do not change unnecessarily
> # weight 5.460
> alg straw
> hash 0 # rjenkins1
> item osd.2 weight 2.730
> item osd.3 weight 2.730
> }
> host sn2 {
> id -4 # do not change unnecessarily
> # weight 5.460
> alg straw
> hash 0 # rjenkins1
> item osd.4 weight 2.730
> item osd.5 weight 2.730
> }
> root default {
> id -1 # do not change unnecessarily
> # weight 16.380
> alg straw
> hash 0 # rjenkins1
> item sn0 weight 5.460
> item sn1 weight 5.460
> item sn2 weight 5.460
> }
>
> # rules
> rule data {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule metadata {
> ruleset 1
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule rbd {
> ruleset 2
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
>
> # end crush map
The three nodes are in the same rack, so same power, same switch. Any
disk is as good as any other. My intent, is that a copy of each object
is stored on any of 3 nodes (presently 3 is all we have, but that may
change).
The default "size" for the cluster is set to 3, and this seems to be
reflected in the current "size' if each of the pools. My understanding
of the above is:
- take default: pick the top-level group
- chooseleaf firsn type host: choose {pool-size} hosts, then choose an
OSD on each host at random
- emit: dump the objects at those locations
That was the default, and if my understanding is correct, it'll do what
I'm after.
Regards,
--
Stuart Longland
Systems Engineer
_ ___
\ /|_) | T: +61 7 3535 9619
\/ | \ | 38b Douglas Street F: +61 7 3535 9699
SYSTEMS Milton QLD 4064 http://www.vrt.com.au
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
