Re: Offsite replication scenario

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Jan 11, 2019 at 10:07 PM Brian Topping <brian.topping@xxxxxxxxx> wrote:
Hi all,

I have a simple two-node Ceph cluster that I’m comfortable with the care and feeding of. Both nodes are in a single rack and captured in the attached dump, it has two nodes, only one mon, all pools size 2. Due to physical limitations, the primary location can’t move past two nodes at the present time. As far as hardware, those two nodes are 18-core Xeon with 128GB RAM and connected with 10GbE.

My next goal is to add an offsite replica and would like to validate the plan I have in mind. For it’s part, the offsite replica can be considered read-only except for the occasional snapshot in order to run backups to tape. The offsite location is connected with a reliable and secured ~350Kbps WAN link.

Unfortunately this is just not going to work. All writes to a Ceph OSD are replicated synchronously to every replica, all reads are served from the primary OSD for any given piece of data, and unless you do some hackery on your CRUSH map each of your 3 OSD nodes is going to be a primary for about 1/3 of the total data.

If you want to move your data off-site asynchronously, there are various options for doing that in RBD (either periodic snapshots and export-diff, or by maintaining a journal and streaming it out) and RGW (with the multi-site stuff). But you're not going to be successful trying to stretch a Ceph cluster over that link.
-Greg
 

The following presuppositions bear challenge:

* There is only a single mon at the present time, which could be expanded to three with the offsite location. Two mons at the primary location is obviously a lower MTBF than one, but  with a third one on the other side of the WAN, I could create resiliency against *either* a WAN failure or a single node maintenance event.
* Because there are two mons at the primary location and one at the offsite, the degradation mode for a WAN loss (most likely scenario due to facility support) leaves the primary nodes maintaining the quorum, which is desirable.
* It’s clear that a WAN failure and a mon failure at the primary location will halt cluster access.
* The CRUSH maps will be managed to reflect the topology change.

If that’s a good capture so far, I’m comfortable with it. What I don’t understand is what to expect in actual use:

* Is the link speed asymmetry between the two primary nodes and the offsite node going to create significant risk or unexpected behaviors?
* Will the performance of the two primary nodes be limited to the speed that the offsite mon can participate? Or will the primary mons correctly calculate they have quorum and keep moving forward under normal operation?
* In the case of an extended WAN outage (and presuming full uptime on primary site mons), would return to full cluster health be simply a matter of time? Are there any limits on how long the WAN could be down if the other two maintain quorum?

I hope I’m asking the right questions here. Any feedback appreciated, including blogs and RTFM pointers.


Thanks for a great product!! I’m really excited for this next frontier!

Brian

> [root@gw01 ~]# ceph -s
>  cluster:
>    id:     nnnn
>    health: HEALTH_OK
>
>  services:
>    mon: 1 daemons, quorum gw01
>    mgr: gw01(active)
>    mds: cephfs-1/1/1 up  {0=gw01=up:active}
>    osd: 8 osds: 8 up, 8 in
>
>  data:
>    pools:   3 pools, 380 pgs
>    objects: 172.9 k objects, 11 GiB
>    usage:   30 GiB used, 5.8 TiB / 5.8 TiB avail
>    pgs:     380 active+clean
>
>  io:
>    client:   612 KiB/s wr, 0 op/s rd, 50 op/s wr
>
> [root@gw01 ~]# ceph df
> GLOBAL:
>    SIZE        AVAIL       RAW USED     %RAW USED
>    5.8 TiB     5.8 TiB       30 GiB          0.51
> POOLS:
>    NAME                ID     USED        %USED     MAX AVAIL     OBJECTS
>    cephfs_metadata     2      264 MiB         0       2.7 TiB        1085
>    cephfs_data         3      8.3 GiB      0.29       2.7 TiB      171283
>    rbd                 4      2.0 GiB      0.07       2.7 TiB         542
> [root@gw01 ~]# ceph osd tree
> ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
> -1       5.82153 root default                         
> -3       2.91077     host gw01                         
> 0   ssd 0.72769         osd.0     up  1.00000 1.00000
> 2   ssd 0.72769         osd.2     up  1.00000 1.00000
> 4   ssd 0.72769         osd.4     up  1.00000 1.00000
> 6   ssd 0.72769         osd.6     up  1.00000 1.00000
> -5       2.91077     host gw02                         
> 1   ssd 0.72769         osd.1     up  1.00000 1.00000
> 3   ssd 0.72769         osd.3     up  1.00000 1.00000
> 5   ssd 0.72769         osd.5     up  1.00000 1.00000
> 7   ssd 0.72769         osd.7     up  1.00000 1.00000
> [root@gw01 ~]# ceph osd df
> ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE VAR  PGS
> 0   ssd 0.72769  1.00000 745 GiB 4.9 GiB 740 GiB 0.66 1.29 115
> 2   ssd 0.72769  1.00000 745 GiB 3.1 GiB 742 GiB 0.42 0.82  83
> 4   ssd 0.72769  1.00000 745 GiB 3.6 GiB 742 GiB 0.49 0.96  90
> 6   ssd 0.72769  1.00000 745 GiB 3.5 GiB 742 GiB 0.47 0.93  92
> 1   ssd 0.72769  1.00000 745 GiB 3.4 GiB 742 GiB 0.46 0.90  76
> 3   ssd 0.72769  1.00000 745 GiB 3.9 GiB 741 GiB 0.52 1.02 102
> 5   ssd 0.72769  1.00000 745 GiB 3.9 GiB 741 GiB 0.52 1.02  98
> 7   ssd 0.72769  1.00000 745 GiB 4.0 GiB 741 GiB 0.54 1.06 104
>                    TOTAL 5.8 TiB  30 GiB 5.8 TiB 0.51         
> MIN/MAX VAR: 0.82/1.29  STDDEV: 0.07
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux