Hey Olivier,
On 07/23/2014 02:06 PM, Olivier DELHOMME wrote:
> Hello,
>
> I'm running a test cluster (mon and osd are debian 7
> with 3.2.57-3+deb7u2 kernel). The client is a debian 7
> with a 3.15.4 kernel that I compiled myself.
>
> The cluster has 3 monitors and 16 osd servers.
> I created a pool (periph) and used it a bit and then
> I decided to create some buckets and moved the hosts
> into :
Can you share your crush map?
Cheers!
-Joao
>
> $ ceph osd crush add-bucket rack1-pdu1 rack
> $ ceph osd crush add-bucket rack1-pdu2 rack
> $ ceph osd crush add-bucket rack1-pdu3 rack
> $ ceph osd crush add-bucket rack2-pdu1 rack
> $ ceph osd crush add-bucket rack2-pdu2 rack
> $ ceph osd crush add-bucket rack2-pdu3 rack
> $ ceph osd crush move ceph-osd0 rack=rack1-pdu1
> $ ceph osd crush move ceph-osd1 rack=rack1-pdu1
> $ ceph osd crush move ceph-osd2 rack=rack1-pdu1
> $ ceph osd crush move ceph-osd3 rack=rack1-pdu2
> $ ceph osd crush move ceph-osd4 rack=rack1-pdu2
> $ ceph osd crush move ceph-osd5 rack=rack1-pdu2
> $ ceph osd crush move ceph-osd6 rack=rack1-pdu3
> $ ceph osd crush move ceph-osd7 rack=rack1-pdu3
> $ ceph osd crush move ceph-osd8 rack=rack1-pdu3
> $ ceph osd crush move ceph-osd9 rack=rack2-pdu1
> $ ceph osd crush move ceph-osd10 rack=rack2-pdu1
> $ ceph osd crush move ceph-osd11 rack=rack2-pdu1
> $ ceph osd crush move ceph-osd12 rack=rack2-pdu2
> $ ceph osd crush move ceph-osd13 rack=rack2-pdu2
> $ ceph osd crush move ceph-osd14 rack=rack2-pdu2
> $ ceph osd crush move ceph-osd15 rack=rack2-pdu3
>
> It did well :
>
> $ ceph osd tree
> # id weight type name up/down reweight
> -23 0.91 rack rack2-pdu3
> -17 0.91 host ceph-osd15
> 15 0.91 osd.15 up 1
> -22 1.81 rack rack2-pdu2
> -14 0.45 host ceph-osd12
> 12 0.45 osd.12 up 1
> -15 0.45 host ceph-osd13
> 13 0.45 osd.13 up 1
> -16 0.91 host ceph-osd14
> 14 0.91 osd.14 up 1
> -21 1.35 rack rack2-pdu1
> -11 0.45 host ceph-osd9
> 9 0.45 osd.9 up 1
> -12 0.45 host ceph-osd10
> 10 0.45 osd.10 up 1
> -13 0.45 host ceph-osd11
> 11 0.45 osd.11 up 1
> -20 1.35 rack rack1-pdu3
> -8 0.45 host ceph-osd6
> 6 0.45 osd.6 up 1
> -9 0.45 host ceph-osd7
> 7 0.45 osd.7 up 1
> -10 0.45 host ceph-osd8
> 8 0.45 osd.8 up 1
> -19 1.35 rack rack1-pdu2
> -5 0.45 host ceph-osd3
> 3 0.45 osd.3 up 1
> -6 0.45 host ceph-osd4
> 4 0.45 osd.4 up 1
> -7 0.45 host ceph-osd5
> 5 0.45 osd.5 up 1
> -18 1.35 rack rack1-pdu1
> -2 0.45 host ceph-osd0
> 0 0.45 osd.0 up 1
> -3 0.45 host ceph-osd1
> 1 0.45 osd.1 up 1
> -4 0.45 host ceph-osd2
> 2 0.45 osd.2 up 1
> -1 0 root default
>
>
> But then, when trying to set the crush_ruleset to the
> pool with the command below it crashes two of the three
> monitor.
>
>
> $ ceph osd pool set periph crush_ruleset 2
> 2014-07-23 14:43:38.942811 7fa9696a3700 0 monclient: hunting for new mon
>
> The first monitor come to :
>
> -4> 2014-07-23 14:43:37.476121 7f52d2f46700 1 -- 10.1.2.1:6789/0 --> 10.1.2.100:0/1027991 -- mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 -- ?+29681 0x3b1c780 con 0x2a578c0
> -3> 2014-07-23 14:43:37.598549 7f52d2f46700 1 -- 10.1.2.1:6789/0 <== client.39105 10.1.2.100:0/1027991 8 ==== mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "periph", "val": "2"} v 0) v1 ==== 122+0+0 (2844980124 0 0) 0x3b1d860 con 0x2a578c0
> -2> 2014-07-23 14:43:37.598602 7f52d2f46700 0 mon.ceph-mon0 at 0(leader) e2 handle_command mon_command({"var": "crush_ruleset", "prefix": "osd pool set", "pool": "periph", "val": "2"} v 0) v1
> -1> 2014-07-23 14:43:37.598705 7f52d2f46700 1 mon.ceph-mon0 at 0(leader).paxos(paxos active c 542663..543338) is_readable now=2014-07-23 14:43:37.598708 lease_expire=2014-07-23 14:43:41.683421 has v0 lc 543338
> 0> 2014-07-23 14:43:37.601706 7f52d2f46700 -1 *** Caught signal (Segmentation fault) **
> in thread 7f52d2f46700
>
>
> Then, after an election try the second monitor goes down also :
>
> -1> 2014-07-23 14:43:51.772370 7eff4ba15700 1 mon.ceph-mon1 at 1(leader).paxos(paxos active c 542663..543338) is_readable now=2014-07-23 14:43:51.772373 lease_expire=2014-07-23 14:43:56.770906 has v0 lc 543338
> 0> 2014-07-23 14:43:51.775817 7eff4ba15700 -1 *** Caught signal (Segmentation fault) **
>
> I can not reactivate the monitors until the command
> "ceph osd pool set periph crush_ruleset 2" is running.
>
> When I kill this command, then the monitors can run
> again and retrieve a normal state but it leaves the
> cluster with some warnings about data replacements
> not achieved I guess (I had some data in the pool).
>
> cluster 4a8669b9-b379-43b2-9488-7fca6e1366bc
> health HEALTH_WARN 152 pgs peering; 166 pgs stuck inactive; 620 pgs stuck unclean; recovery 620/694410 objects degraded (0.089%)
> monmap e2: 3 mons at {ceph-mon0=10.1.2.1:6789/0,ceph-mon1=10.1.2.2:6789/0,ceph-mon2=10.1.2.3:6789/0}, election epoch 50, quorum 0,1,2 ceph-mon0,ceph-mon1,ceph-mon2
> osdmap e1688: 16 osds: 16 up, 16 in
> pgmap v300875: 712 pgs, 5 pools, 1350 GB data, 338 kobjects
> 2765 GB used, 5614 GB / 8379 GB avail
> 620/694410 objects degraded (0.089%)
> 14 inactive
> 152 peering
> 454 active+remapped
> 92 active+clean
>
> Is there something that I did wrong or forgot to do ?
>
> While writing this mail down I realised that there is
> only one host in the rack2-pdu3 rack. May this be a
> cause of the problem ?
>
> Thanks for any hints.
>
--
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
