Re: Porblem creating Geo-replication

wodel youchi <wodel_doom@xxxxxxxx> · Mon, 15 Dec 2014 22:46:01 +0000 (UTC)

Hi,

I restarted my LAB from scratch, new Glusterfs VM servers.

This time:
 I had the same problem with the locale variables.
The creation of the geo-replication did work and I did add the command="/usr/libexec/glusterfs/gsyncd" to the authorized_keys file of the geoaccount user.

but, the existing dir/files on the master volume were not synced, only new ones.
and no, the slave was empty at the beginning of the test.

Regards

     Le Lundi 15 décembre 2014 7h10, Aravinda <avishwan@xxxxxxxxxx> a écrit :

    On 12/14/2014 02:57 AM, wodel youchi
      wrote:

      Hi again,

        I had a hard time to configure geo-replication. After creating
        the session, the start gave me faulty state and I had to deal
        with it, it was the /nonexistent/gsyncd in
        /var/lib/glusterd/geo-replication/data1_node3.example_data2/gsyncd.conf
        to change to /usr/libexec/gluster/gsyncd

    When you get /nonexistent/gsyncd, that means issue with pem keys
    setup. For security reason while pushing the pem keys to slave, we
    add command=<COMMAND TO RUN> before key in
    /root/.ssh/authorized_keys.

    For example,

    command="/usr/local/libexec/glusterfs/gsyncd"  ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqxqMiZ8dyXUQq0pLVOYpRSsC+aYFn6pbPQZ3LtRPKGYfA63SNoYni
fhnM2UR9fnZz3hisBUxIzcVrVux2y3ojI/vPFFi08tVtK8/r9yoqqh3HqlBnotY50H1/1qeco+71U9hy276fUONP64KoOZtme3MwYuoNz
4z1NvCQFcEbXtPfHO5A9P3C+NuMhgNK8N63RSCzZ6dtO+wZygbVJlbPNQxp8Y5E8rbIuzRy6bD/0nmEKc/nqvEYTYgkck
ES0Xy92JVxbcwCOnZFNi4rT6+HarDIuFRB835I5ss+QBrT9SM09qmFuQ== root@fedoravm1

If you have any other entry in /root/.ssh/authorized_keys file of slave machine with same key then you will get /nonexistent/gsyncd error. 

Advantage of having "command" in authorized key is, if any master node is compromised, 
then they can login to any slave node. But with this command option, logged in user is 
only limited to use gsyncd or the command specified in authorized_keys.

        after that the geo-started, but nothing happened, no file or
        directory was synced.

        the gluster volume geo-replication data1
        geoaccount@xxxxxxxxxxxxxxxxx::data2 status

        keeps saying: Changelog Crawl

        [2014-12-13 21:57:17.129836] W
        [master(/mnt/srv1/brick1):1005:process] _GMaster: incomplete
        sync, retrying changelogs: CHANGELOG.1418504186

        [2014-12-13 21:57:22.648163] W
        [master(/mnt/srv1/brick1):294:regjob] _GMaster: Rsync:
        .gfid/f066ca4a-2d31-4342-bc7e-a37da25b2253 [errcode: 23]

        [2014-12-13 21:57:22.648426] W
        [master(/mnt/srv1/brick1):986:process] _GMaster: changelogs
        CHANGELOG.1418504186 could not be processed - moving on...

        but new files/directories are synced, so I deleted all data on
        the master cluster and recreated them, and all was synced

        but the status command (above) keeps saying Changelog Crawl

        On the slave node I have these logs

        [2014-12-13 21:11:12.149041] W
        [client-rpc-fops.c:1210:client3_3_removexattr_cbk]
        0-data2-client-0: remote operation failed: No data available

        [2014-12-13 21:11:12.149067] W [fuse-bridge.c:1261:fuse_err_cbk]
        0-glusterfs-fuse: 3243: REMOVEXATTR()
        /.gfid/cd26bcc2-9b9f-455a-a2b2-9b6358f24203 => -1 (No data
        available)

        [2014-12-13 21:11:12.516674] W
        [client-rpc-fops.c:1210:client3_3_removexattr_cbk]
        0-data2-client-0: remote operation failed: No data available

        [2014-12-13 21:11:12.516705] W [fuse-bridge.c:1261:fuse_err_cbk]
        0-glusterfs-fuse: 3325: REMOVEXATTR()
        /.gfid/e6142b37-2362-4c95-a291-f396a122b014 => -1 (No data
        available)

        [2014-12-13 21:11:12.517577] W
        [client-rpc-fops.c:1210:client3_3_removexattr_cbk]
        0-data2-client-0: remote operation failed: No data available

        [2014-12-13 21:11:12.517600] W [fuse-bridge.c:1261:fuse_err_cbk]
        0-glusterfs-fuse: 3331: REMOVEXATTR()
        /.gfid/e6142b37-2362-4c95-a291-f396a122b014 => -1 (No data
        available)

        and

        [2014-12-13 21:57:16.741321] W
        [syncdutils(slave):480:errno_wrap] <top>: reached maximum
        retries (['.gfid/ba9c75ef-d4f7-4a6b-923f-82a8c7be4443',
        'glusterfs.gfid.newfile',
'\x00\x00\x00\x1b\x00\x00\x00\x1bcab3ae81-7b52-4c55-ac33-37814ff374c4\x00\x00\x00\x81\xb0glustercli1.lower-test\x00\x00\x00\x01\xb0\x00\x00\x00\x00\x00\x00\x00\x00'])...

        [2014-12-13 21:57:22.400269] W
        [syncdutils(slave):480:errno_wrap] <top>: reached maximum
        retries (['.gfid/ba9c75ef-d4f7-4a6b-923f-82a8c7be4443',
        'glusterfs.gfid.newfile',
'\x00\x00\x00\x1b\x00\x00\x00\x1bcab3ae81-7b52-4c55-ac33-37814ff374c4\x00\x00\x00\x81\xb0glustercli1.lower-test\x00\x00\x00\x01\xb0\x00\x00\x00\x00\x00\x00\x00\x00'])...

        I didn't find what does this mean

    Does your slave had data before Geo-replication session is created.
    From the log what I can see is, geo-rep is failing to create a file
    in slave(may be same file name exists with different GFID(GFID is
    GlusterFS unique identifier for file))

    Rsync is failing since file is not created in slave, and it is
    unable to sync.

    --

    regards

    Aravinda

      any idea.

        Regards

                Le Vendredi
                  12 décembre 2014 19h35, wodel youchi
                  <wodel_doom@xxxxxxxx> a écrit :

                      Thanks
                          for your reply,

                      When executing the gverify.sh
                        script I had these errors on slave.log
                      [2014-12-12
                        18:12:45.423669] I
                        [options.c:1163:xlator_option_init_double]
                        0-fuse: option attribute-timeout convertion
                        failed value 1.0

                        [2014-12-12 18:12:45.423689] E
                        [xlator.c:425:xlator_init] 0-fuse:
                        Initialization of volume 'fuse' failed, review
                        your volfile again

                      I think that the
                          problem is linked to the locale variables,
                          mine were

                      LANG=fr_FR.UTF-8

                          LC_CTYPE="fr_FR.UTF-8"

                      LC_NUMERIC="fr_FR.UTF-8"

                      LC_TIME="fr_FR.UTF-8"

                          LC_COLLATE="fr_FR.UTF-8"

                          LC_MONETARY="fr_FR.UTF-8"

                          LC_MESSAGES="fr_FR.UTF-8"

                          LC_PAPER="fr_FR.UTF-8"

                          LC_NAME="fr_FR.UTF-8"

                          LC_ADDRESS="fr_FR.UTF-8"

                          LC_TELEPHONE="fr_FR.UTF-8"

                          LC_MEASUREMENT="fr_FR.UTF-8"

                          LC_IDENTIFICATION="fr_FR.UTF-8"

                          LC_ALL=

                      I changed LC_CTYPE and LC_NUMERIC to C and then
                          executed the gverify.sh script again and it
                          worked, but the gluster vol geo-rep ...
                          failed.

                      I then changed the /etc/locale.conf
                          file and modified the LANG from fr_FR.UTF-8 to C, rebooted
                          the VM and voila, the geo-replication session
                          was created successfuly.

                      but
                        I am not sure if my changes won't affect other
                        things.

                      Regards

                                Le Vendredi 12 décembre 2014
                                  8h30, Kotresh Hiremath Ravishankar
                                  <khiremat@xxxxxxxxxx> a écrit :

                              Hi,

                                The setup is failing while doing
                                compatibility test between master and
                                slave cluster.

                                The gverify.sh script is failing to get
                                master volume details for the same.

                                Could you run the following and paste
                                the output here?

                                bash -x
                                /usr/local/libexec/glusterfs/gverify.sh
                                <master-vol-name> root
                                <slave-host-name>
                                <slave-vol-name>
                                <temp-log-file>

                                If source installed gverify.sh is found
                                in above path where as if rpm install,

                                it is found in
                                /usr/libexec/glusterfs/gverify.sh

                                If you are sure the master and slave
                                gluster versions and size is fine, the
                                easy workaround

                                is to use force.

                                gluster vol geo-replication
                                <master-vol>
                                <slave-host>::<slave-vol>
                                create push-pem force

                                Thanks and Regards,

                                Kotresh H R

                                  ----- Original Message -----

                                  From: "wodel youchi" <wodel_doom@xxxxxxxx>

                                  To: gluster-users@xxxxxxxxxxx

                                  Sent: Friday, December 12, 2014
                                  3:13:48 AM

                                  Subject:  Porblem
                                  creating Geo-replication

                                  Hi, 

                                  I am using Centos7x64 updates 

                                  GlusterFS 3.6 from http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/epel-7Server/
                                  repository. 

                                  No firewall and No Selinux. 

                                  I've two nodes with distributed
                                  replicated volume: data1 

                                  and a third node with a distributed
                                  volume: data2 

                                  the two volumes have the same size 

                                  I've trouble to configure
                                  geo-replication to the third node,
                                  I've been following the RedHat Storage
                                  3 Admin Guide, but It does not work. 

                                  I've created the ssh-passwordless
                                  connection between the nodes, and
                                  followed these commands 

                                  On the master: 

                                  # gluster system: : execute
                                  gsec_create 

                                  Common secret pub file present at
                                  /var/lib/glusterd/geo-replication/common_secret.pem.pub

                                  # gluster volume geo-replication data1
                                  node3.example.com::data2 create
                                  push-pem 

                                  Unable to fetch master volume details.
                                  Please check the master cluster and
                                  master volume. 

                                  geo-replication command failed 

                                  from
                                  /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
                                  I get these error messges 

                                  [2014-12-11 21:34:47.152644] E
                                  [glusterd-geo-rep.c:2012:glusterd_verify_slave]
                                  0-: Not a valid slave 

                                  [2014-12-11 21:34:47.152750] E
                                  [glusterd-geo-rep.c:2240:glusterd_op_stage_gsync_create]
                                  0-: node3.example.com::data2 is not a
                                  valid slave volume. Error: Unable to
                                  fetch master volume details. Please
                                  check the master cluster and master
                                  volume. 

                                  [2014-12-11 21:34:47.152764] E
                                  [glusterd-syncop.c:1151:gd_stage_op_phase]
                                  0-management: Staging of operation
                                  'Volume Geo-replication Create' failed
                                  on localhost : Unable to fetch master
                                  volume details. Please check the
                                  master cluster and master volume. 

                                  [2014-12-11 21:35:25.559144] E
                                  [glusterd-handshake.c:914:gd_validate_mgmt_hndsk_req]
                                  0-management: Rejecting management
                                  handshake request from unknown peer
                                  192.168.1.9:1005 

                                  the 192.168.1.9 is the IP address of
                                  the 3rd node. 

                                  any idea!!. 

                                  thanks 

_______________________________________________

                                  Gluster-users mailing list

                                  Gluster-users@xxxxxxxxxxx

                                  http://supercolony.gluster.org/mailman/listinfo/gluster-users

                _______________________________________________

                  Gluster-users mailing list

                  Gluster-users@xxxxxxxxxxx

                  http://supercolony.gluster.org/mailman/listinfo/gluster-users

      _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users