| Hi everyone, Beginner with Ceph, i’m looking for a way to do a 3-way replication between 2 datacenters as mention in ceph docs (but not describe). My goal is to keep access to the data (at least read-only access) even when the link between the 2 datacenters is cut and make sure at least one copy of the data exists in each datacenter. I’m not sure how to implement such 3-way replication. With a rule? Based on the CEPH docs, I think of a rule: rule 3-way-replication_with_2_DC { ruleset 1 type replicated min_size 2 max_size 3 step take DC-1 step choose firstn 1 type host step chooseleaf firstn 1 type osd step emit step take DC-2 step choose firstn 1 type host step chooseleaf firstn 1 type osd step emit step take default step choose firstn 1 type host step chooseleaf firstn 1 type osd step emit } but what should happen if the link between the 2 datacenters is cut? If someone has a better solution, I interested by any resources about it (examples, …). The default rule (see below) keep the pool working when we mark each node of DC-2 as down (typically maintenance) but if we shut the link down between the 2 datacenters, the pool/rbd hangs (frozen writing dd tool for example). Does anyone have some insight on how to setup a 3-way replication between 2 datacenters? Thanks in advance for any advice on the topic. Current situation: Mons : host-1, host-2, host-4 Quick network topology: USERS NETWORK | 2x10G | DC-1-SWITCH <——— 40G ——> DC-2-SWITCH | | | | | | host-1 _| | | host-4 _| | | host-2 ___| | host-5 ___| | host-3 _____| host-6 _____| crushmap : # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 147.33325 root default -20 73.66663 datacenter DC-1 -15 73.66663 rack DC-1-RACK-1 -9 24.55554 host host-1 27 hdd 2.72839 osd.27 up 1.00000 1.00000 28 hdd 2.72839 osd.28 up 1.00000 1.00000 29 hdd 2.72839 osd.29 up 1.00000 1.00000 30 hdd 2.72839 osd.30 up 1.00000 1.00000 31 hdd 2.72839 osd.31 up 1.00000 1.00000 32 hdd 2.72839 osd.32 up 1.00000 1.00000 33 hdd 2.72839 osd.33 up 1.00000 1.00000 34 hdd 2.72839 osd.34 up 1.00000 1.00000 36 hdd 2.72839 osd.36 up 1.00000 1.00000 -11 24.55554 host host-2 35 hdd 2.72839 osd.35 up 1.00000 1.00000 37 hdd 2.72839 osd.37 up 1.00000 1.00000 38 hdd 2.72839 osd.38 up 1.00000 1.00000 39 hdd 2.72839 osd.39 up 1.00000 1.00000 40 hdd 2.72839 osd.40 up 1.00000 1.00000 41 hdd 2.72839 osd.41 up 1.00000 1.00000 42 hdd 2.72839 osd.42 up 1.00000 1.00000 43 hdd 2.72839 osd.43 up 1.00000 1.00000 46 hdd 2.72839 osd.46 up 1.00000 1.00000 -13 24.55554 host host-3 44 hdd 2.72839 osd.44 up 1.00000 1.00000 45 hdd 2.72839 osd.45 up 1.00000 1.00000 47 hdd 2.72839 osd.47 up 1.00000 1.00000 48 hdd 2.72839 osd.48 up 1.00000 1.00000 49 hdd 2.72839 osd.49 up 1.00000 1.00000 50 hdd 2.72839 osd.50 up 1.00000 1.00000 51 hdd 2.72839 osd.51 up 1.00000 1.00000 52 hdd 2.72839 osd.52 up 1.00000 1.00000 53 hdd 2.72839 osd.53 up 1.00000 1.00000 -19 73.66663 datacenter DC-2 -16 73.66663 rack DC-2-RACK-1 -3 24.55554 host host-4 0 hdd 2.72839 osd.0 up 1.00000 1.00000 1 hdd 2.72839 osd.1 up 1.00000 1.00000 2 hdd 2.72839 osd.2 up 1.00000 1.00000 3 hdd 2.72839 osd.3 up 1.00000 1.00000 4 hdd 2.72839 osd.4 up 1.00000 1.00000 5 hdd 2.72839 osd.5 up 1.00000 1.00000 6 hdd 2.72839 osd.6 up 1.00000 1.00000 7 hdd 2.72839 osd.7 up 1.00000 1.00000 8 hdd 2.72839 osd.8 up 1.00000 1.00000 -5 24.55554 host host-5 9 hdd 2.72839 osd.9 up 1.00000 1.00000 10 hdd 2.72839 osd.10 up 1.00000 1.00000 11 hdd 2.72839 osd.11 up 1.00000 1.00000 12 hdd 2.72839 osd.12 up 1.00000 1.00000 13 hdd 2.72839 osd.13 up 1.00000 1.00000 14 hdd 2.72839 osd.14 up 1.00000 1.00000 15 hdd 2.72839 osd.15 up 1.00000 1.00000 16 hdd 2.72839 osd.16 up 1.00000 1.00000 18 hdd 2.72839 osd.18 up 1.00000 1.00000 -7 24.55554 host host-6 19 hdd 2.72839 osd.19 up 1.00000 1.00000 20 hdd 2.72839 osd.20 up 1.00000 1.00000 21 hdd 2.72839 osd.21 up 1.00000 1.00000 22 hdd 2.72839 osd.22 up 1.00000 1.00000 23 hdd 2.72839 osd.23 up 1.00000 1.00000 24 hdd 2.72839 osd.24 up 1.00000 1.00000 25 hdd 2.72839 osd.25 up 1.00000 1.00000 26 hdd 2.72839 osd.26 up 1.00000 1.00000 54 hdd 2.72839 osd.54 up 1.00000 1.00000 current rules (default one) : # ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
Cordialement / Best regards,
Sébastien VIGNERON CRIANN, Ingénieur / Engineer Technopôle du Madrillet 745, avenue de l'Université 76800 Saint-Etienne du Rouvray - France tél. +33 2 32 91 42 91 fax. +33 2 32 91 42 92 http://www.criann.fr mailto:sebastien.vigneron@xxxxxxxxx support: support@xxxxxxxxx |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
