Re: Needed help to setup a 3-way replication between 2 datacenters

[Peter Linder <peter.linder@xxxxxxxxxxxxxx>]

On 11/10/2017 7:17 AM, Sébastien VIGNERON wrote:

Hi everyone,

Beginner with Ceph, i’m looking for a way to do a 3-way replication between 2 datacenters as mention in ceph docs (but not describe).

My goal is to keep access to the data (at least read-only access) even when the link between the 2 datacenters is cut and make sure at least one copy of the data exists in each datacenter.

If that is your goal, then why 3 way replication?

I’m not sure how to implement such 3-way replication. With a rule?

Based on the CEPH docs, I think of a rule:

rule 3-way-replication_with_2_DC {

ruleset 1

type replicated

min_size 2

max_size 3

min_size and max_size here doesn't do what you expect it to. You need to set min_size = 1 for a 2 way replicated cluster (beware of inconcistencies if the link between DC's go down) and = 2 for a 3 way replicated cluster, but the setting is on the pool, not in the crush rule.

step take DC-1

step choose firstn 1 type host

step chooseleaf firstn 1 type osd

step emit

step take DC-2

step choose firstn 1 type host

step chooseleaf firstn 1 type osd

step emit

step take default

step choose firstn 1 type host

step chooseleaf firstn 1 type osd

step emit

}

but what should happen if the link between the 2 datacenters is cut? If someone has a better solution, I interested by any resources about it (examples, …).

This seems to, for each pg, take an osd on a host in DC-1 and then and osd on a host in DC-2, and then just a random osd on a random host anywhere. 50% of the extra osds selected will be in DC1 and the rest in DC2. When the link is cut, 50% of the pgs will not be able to fulfil the min_size = 2 requirement (depending on if the observer is in DC1 or DC2 of course) and operations will stop on these. This should in practice stop all operations, and Im not even considering monitor quorum yet.

The default rule (see below) keep the pool working when we mark each node of DC-2 as down (typically maintenance) but if we shut the link down between the 2 datacenters, the pool/rbd hangs (frozen writing dd tool for example).

Does anyone have some insight on how to setup a 3-way replication between 2 datacenters?

I don't really know why there is a difference here. We opted for a 3 way cluster in 3 separate datacenters though. Perhaps somehow you can simulate 2 separate datacenters in one of yours, at least make sure they are on different power circuits etc. Also, consider redundancy for your network so that is does not go down. Spanning tree is a little slow, but TRILL or SPB should work in your case.

Thanks in advance for any advice on the topic.

Current situation:

Mons : host-1, host-2, host-4

Quick network topology:

USERS NETWORK

     |

   2x10G

     |

  DC-1-SWITCH <——— 40G ——> DC-2-SWITCH

        | | |                   | | |

host-1 _| | |           host-4 _| | |

host-2 ___| |           host-5 ___| |

host-3 _____|           host-6 _____|

crushmap :

# ceph osd tree

ID  CLASS WEIGHT    TYPE NAME                   STATUS REWEIGHT PRI-AFF

 -1       147.33325 root default

-20        73.66663     datacenter DC-1

-15        73.66663         rack DC-1-RACK-1

 -9        24.55554             host host-1

 27   hdd   2.72839                 osd.27          up  1.00000 1.00000

 28   hdd   2.72839                 osd.28          up  1.00000 1.00000

 29   hdd   2.72839                 osd.29          up  1.00000 1.00000

 30   hdd   2.72839                 osd.30          up  1.00000 1.00000

 31   hdd   2.72839                 osd.31          up  1.00000 1.00000

 32   hdd   2.72839                 osd.32          up  1.00000 1.00000

 33   hdd   2.72839                 osd.33          up  1.00000 1.00000

 34   hdd   2.72839                 osd.34          up  1.00000 1.00000

 36   hdd   2.72839                 osd.36          up  1.00000 1.00000

-11        24.55554             host host-2

 35   hdd   2.72839                 osd.35          up  1.00000 1.00000

 37   hdd   2.72839                 osd.37          up  1.00000 1.00000

 38   hdd   2.72839                 osd.38          up  1.00000 1.00000

 39   hdd   2.72839                 osd.39          up  1.00000 1.00000

 40   hdd   2.72839                 osd.40          up  1.00000 1.00000

 41   hdd   2.72839                 osd.41          up  1.00000 1.00000

 42   hdd   2.72839                 osd.42          up  1.00000 1.00000

 43   hdd   2.72839                 osd.43          up  1.00000 1.00000

 46   hdd   2.72839                 osd.46          up  1.00000 1.00000

-13        24.55554             host host-3

 44   hdd   2.72839                 osd.44          up  1.00000 1.00000

 45   hdd   2.72839                 osd.45          up  1.00000 1.00000

 47   hdd   2.72839                 osd.47          up  1.00000 1.00000

 48   hdd   2.72839                 osd.48          up  1.00000 1.00000

 49   hdd   2.72839                 osd.49          up  1.00000 1.00000

 50   hdd   2.72839                 osd.50          up  1.00000 1.00000

 51   hdd   2.72839                 osd.51          up  1.00000 1.00000

 52   hdd   2.72839                 osd.52          up  1.00000 1.00000

 53   hdd   2.72839                 osd.53          up  1.00000 1.00000

-19        73.66663     datacenter DC-2

-16        73.66663         rack DC-2-RACK-1

 -3        24.55554             host host-4

  0   hdd   2.72839                 osd.0           up  1.00000 1.00000

  1   hdd   2.72839                 osd.1           up  1.00000 1.00000

  2   hdd   2.72839                 osd.2           up  1.00000 1.00000

  3   hdd   2.72839                 osd.3           up  1.00000 1.00000

  4   hdd   2.72839                 osd.4           up  1.00000 1.00000

  5   hdd   2.72839                 osd.5           up  1.00000 1.00000

  6   hdd   2.72839                 osd.6           up  1.00000 1.00000

  7   hdd   2.72839                 osd.7           up  1.00000 1.00000

  8   hdd   2.72839                 osd.8           up  1.00000 1.00000

 -5        24.55554             host host-5

  9   hdd   2.72839                 osd.9           up  1.00000 1.00000

 10   hdd   2.72839                 osd.10          up  1.00000 1.00000

 11   hdd   2.72839                 osd.11          up  1.00000 1.00000

 12   hdd   2.72839                 osd.12          up  1.00000 1.00000

 13   hdd   2.72839                 osd.13          up  1.00000 1.00000

 14   hdd   2.72839                 osd.14          up  1.00000 1.00000

 15   hdd   2.72839                 osd.15          up  1.00000 1.00000

 16   hdd   2.72839                 osd.16          up  1.00000 1.00000

 18   hdd   2.72839                 osd.18          up  1.00000 1.00000

 -7        24.55554             host host-6

 19   hdd   2.72839                 osd.19          up  1.00000 1.00000

 20   hdd   2.72839                 osd.20          up  1.00000 1.00000

 21   hdd   2.72839                 osd.21          up  1.00000 1.00000

 22   hdd   2.72839                 osd.22          up  1.00000 1.00000

 23   hdd   2.72839                 osd.23          up  1.00000 1.00000

 24   hdd   2.72839                 osd.24          up  1.00000 1.00000

 25   hdd   2.72839                 osd.25          up  1.00000 1.00000

 26   hdd   2.72839                 osd.26          up  1.00000 1.00000

 54   hdd   2.72839                 osd.54          up  1.00000 1.00000

current rules (default one) :

# ceph osd crush rule dump

[

    {

        "rule_id": 0,

        "rule_name": "replicated_rule",

        "ruleset": 0,

        "type": 1,

        "min_size": 1,

        "max_size": 10,

        "steps": [

            {

                "op": "take",

                "item": -1,

                "item_name": "default"

            },

            {

                "op": "chooseleaf_firstn",

                "num": 0,

                "type": "host"

            },

            {

                "op": "emit"

            }

        ]

    }

]

Cordialement / Best regards,

Sébastien VIGNERON 
CRIANN, 
Ingénieur / Engineer
Technopôle du Madrillet 
745, avenue de l'Université 
76800 Saint-Etienne du Rouvray - France 
tél. +33 2 32 91 42 91 
fax. +33 2 32 91 42 92 
http://www.criann.fr 
mailto:sebastien.vigneron@xxxxxxxxx
support: support@xxxxxxxxx

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com