Re: Geo-replication

Aravinda VK <aravinda@xxxxxxxxx> · Mon, 2 Mar 2020 13:23:41 +0530

Looks like setup issue to me. Copying SSH keys manually is not required. 
Command prefix is required while adding to authorized_keys file in each remote nodes. That will not be available if ssh keys are added manually.

Geo-rep specifies /nonexisting/gsyncd in the command to make sure it connects via the actual command specified in authorized_keys file, in your case Geo-replication is actually looking for gsyncd command in /nonexisting/gsyncd path.

Please try with push-pem option during Geo-rep create command.

—
regards
Aravinda Vishwanathapura
https://kadalu.io

On 02-Mar-2020, at 6:03 AM, David Cunningham <dcunningham@xxxxxxxxxxxxx> wrote:

Hello,

We've set up geo-replication but it isn't actually syncing. Scenario is that we have two GFS clusters. Cluster A has nodes cafs10, cafs20, and cafs30, replicating with each other over a LAN. Cluster B has nodes nvfs10, nvfs20, and nvfs30 also replicating with each other over a LAN. We are geo-replicating data from the A cluster to the B cluster over the internet. SSH key access is set up, allowing all the A nodes password-less access to root on nvfs10 

Geo-replication was set up using these commands, run on cafs10:

gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 create ssh-port 8822 no-verify
gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 config remote-gsyncd /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd
gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 start

However after a very short period of the status being "Initializing..." the status then sits on "Passive":

# gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 status 
MASTER NODE    MASTER VOL    MASTER BRICK                        SLAVE USER    SLAVE                         SLAVE NODE      STATUS     CRAWL STATUS    LAST_SYNCED          
------------------------------------------------------------------------------------------------------------------------------------------------------------------
cafs10         gvol0         /nodirectwritedata/gluster/gvol0    root          nvfs10.example.com::gvol0    nvfs30.local    Passive    N/A             N/A                  
cafs30         gvol0         /nodirectwritedata/gluster/gvol0    root          nvfs10.example.com::gvol0    N/A             Created    N/A             N/A                  
cafs20         gvol0         /nodirectwritedata/gluster/gvol0    root          nvfs10.example.com::gvol0    N/A             Created    N/A             N/A     

So my questions are:
1. Why does the status on cafs10 mention "nvfs30.local"? That's the LAN address that nvfs10 replicates with nvfs30 using. It's not accessible from the A cluster, and I didn't use it when configuring geo-replication.
2. Why does geo-replication sit in Passive status?

Thanks very much for any assistance.

On Tue, 25 Feb 2020 at 15:46, David Cunningham <dcunningham@xxxxxxxxxxxxx> wrote:
Hi Aravinda and Sunny,

Thank you for the replies. We have 3 replicating nodes on the master side, and want to geo-replicate their data to the remote slave side. As I understand it if the master node which had the geo-replication create command run goes down then another node will take over pushing updates to the remote slave. Is that right?

We have already taken care of adding all master node's SSH keys to the remote slave's authorized_keys externally, so won't include the push-pem part of the create command.

Mostly I wanted to confirm the geo-replication behaviour on the replicating master nodes if one of them goes down. 

Thank you!

On Tue, 25 Feb 2020 at 14:32, Aravinda VK <aravinda@xxxxxxxxx> wrote:
Hi David,

On 25-Feb-2020, at 3:45 AM, David Cunningham <dcunningham@xxxxxxxxxxxxx> wrote:

Hello,

I've a couple of questions on geo-replication that hopefully someone can help with:

1. If there are multiple nodes in a cluster on the master side (pushing updates to the geo-replication slave), which node actually does the pushing? Does GlusterFS decide itself automatically?

Once Geo-replication session is started, one worker will be started corresponding to each Master bricks. Each worker identifies the changes happened in respective brick and sync those changes via Mount. This way load is distributed among Master nodes. In case of Replica sub volume, one worker among the Replica group will become active and participate in the syncing. Other bricks in that Replica group will remain Passive. Passive worker will become Active if the previously Active brick goes down (This is because all Replica bricks will have the same set of changes, syncing from each worker is redundant).

2.With regard to copying SSH keys, presumably the SSH key of all master nodes should be authorized on the geo-replication client side?

Geo-replication session is established between one master node and one remote node. If Geo-rep create command is successful then,

- SSH keys generated in all master nodes
- Public keys from all master nodes are copied to initiator Master node
- Public keys copied to the Remote node specified in the create command
- Master public keys are distributed to all nodes of remote Cluster and added to respective ~/.ssh/authorized_keys

After successful Geo-rep create command, any Master node can connect to any remote node via ssh.

Security: Command prefix is added while adding public key to remote node’s authorized_keys file, So that if anyone gain access using this key can access only gsyncd command.

```
command=gsyncd ssh-key….
```

Thanks for your help.

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
________

Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

—
regards
Aravinda Vishwanathapura
https://kadalu.io

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782

________

Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users