Hello world, I'm currently setting up a few Vagrant boxes to test out geo-replication. I have three boxes in two locations, fra-node[1-3] and chi-node[1-3]. I set up a single replicated volume per cluster (fra-volume and chi-volume) and then initiate a geo-replication from fra to chi. After setting up the mountbroker on one of the slave nodes, when restaring glusterd, it re-creates the file gsyncd_template.conf with root:root ownership: #v+ [root@chi-node2 ~]# chgrp geogroup /var/lib/glusterd/geo-replication/gsyncd_template.conf [root@chi-node2 ~]# chmod g+rw !$ chmod g+rw /var/lib/glusterd/geo-replication/gsyncd_template.conf [root@chi-node2 ~]# ls -l !$ ls -l /var/lib/glusterd/geo-replication/gsyncd_template.conf -rwxrwxr-x 1 root geogroup 1858 Nov 17 10:28 /var/lib/glusterd/geo-replication/gsyncd_template.conf [root@chi-node2 ~]# service glusterd restart Stopping glusterd: [ OK ] Starting glusterd: [ OK ] [root@chi-node2 ~]# ls -l /var/lib/glusterd/geo-replication/gsyncd_template.conf -rwxr-xr-x 1 root root 1858 Nov 17 12:26 /var/lib/glusterd/geo-replication/gsyncd_template.conf #v- And geo-replication fails then fails: #v+ Popen: ssh> [2016-11-17 09:33:56.824232] I [cli.c:730:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.5 Popen: ssh> [2016-11-17 09:33:56.824321] I [cli.c:613:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost Popen: ssh> [2016-11-17 09:33:56.907702] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 Popen: ssh> [2016-11-17 09:33:56.908634] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now Popen: ssh> [2016-11-17 09:33:56.908747] I [cli-rpc-ops.c:6655:gf_cli_getwd_cbk] 0-cli: Received resp to getwd Popen: ssh> [2016-11-17 09:33:56.908833] I [input.c:31:cli_batch] 0-: Exiting with: 0 Popen: ssh> [2016-11-17 09:33:56.985793] E [syncdutils:279:log_raise_exception] <top>: FAIL: Popen: ssh> Traceback (most recent call last): Popen: ssh> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 203, in main Popen: ssh> main_i() Popen: ssh> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 542, in main_i Popen: ssh> upgrade_config_file(rconf['config_file'], confdata) Popen: ssh> File "/usr/libexec/glusterfs/python/syncdaemon/configinterface.py", line 158, in upgrade_config_file Popen: ssh> shutil.move(tempConfigFile.name, path) Popen: ssh> File "/usr/lib64/python2.6/shutil.py", line 260, in move Popen: ssh> copy2(src, real_dst) Popen: ssh> File "/usr/lib64/python2.6/shutil.py", line 95, in copy2 Popen: ssh> copyfile(src, dst) Popen: ssh> File "/usr/lib64/python2.6/shutil.py", line 51, in copyfile Popen: ssh> with open(dst, 'wb') as fdst: Popen: ssh> IOError: [Errno 13] Permission denied: '/var/lib/glusterd/geo-replication/gsyncd_template.conf' Popen: ssh> failed with IOError. #v- This seems related to: https://bugzilla.redhat.com/show_bug.cgi?id=1339683 Involved software versions: glusterfs-3.8.5-1.el6.x86_64 glusterfs-api-3.8.5-1.el6.x86_64 glusterfs-cli-3.8.5-1.el6.x86_64 glusterfs-client-xlators-3.8.5-1.el6.x86_64 glusterfs-fuse-3.8.5-1.el6.x86_64 glusterfs-geo-replication-3.8.5-1.el6.x86_64 glusterfs-libs-3.8.5-1.el6.x86_64 glusterfs-server-3.8.5-1.el6.x86_64 Steps leading up to this: 1. On all nodes (I probably don't need to setup all this on all nodes, but it shouldn't hurt, and make it easier if I ever (I probably don't need to setup all this on all nodes, but it shouldn't hurt, and make it easier if I ever want to switch geoip replication direction): # install packages yum -q -y install glusterfs glusterfs-fuse glusterfs-server \ glusterfs-geo-replication # add geoaccount:geogroup groupadd -r geogroup useradd -r -g geogroup -s /bin/bash -m geoaccount mkdir /home/geoaccount/.ssh # setup passwordless SSH cp /vagrant/geoaccount_key.pub /home/geoaccount/.ssh/authorized_keys chmod -R go-rwx /home/geoaccount/.ssh chown -R geoaccount. /home/geoaccount/.ssh mkdir /root/.ssh cp /vagrant/geoaccount_key /root/.ssh/id_rsa chmod -R go-rwx /root/.ssh # add mountbroker directory mkdir /var/mountbroker-root chmod 0711 /var/mountbroker-root # geo-replication directories and permissions mkdir -p /var/log/glusterfs/geo-replication-slaves /var/lib/glusterd/geo-replication chgrp -R geogroup /var/log/glusterfs/geo-replication-slaves chgrp -R geogroup /var/lib/glusterd/geo-replication chmod -R 770 /var/lib/glusterd/geo-replication chmod -R 770 /var/log/glusterfs/geo-replication-slaves 2. On the fra-node3 and chi-node3 servers ($area is either "fra" or "chi"): gluster peer probe ${area}-node1 gluster peer probe ${area}-node2 gluster volume create ${area}-volume replica $count transport tcp $nodes gluster volume start ${area}-volume gluster volume set all cluster.enable-shared-storage enable 3. On the chi-node3 server: gluster system:: execute mountbroker opt mountbroker-root /var/mountbroker-root gluster system:: execute mountbroker user geoaccount ${area}-volume gluster system:: execute mountbroker opt geo-replication-log-group geogroup gluster system:: execute mountbroker opt rpc-auth-allow-insecure on 4. On all chi-node* servers (since those are replication targets), after settting up the mount broker: service glusterd restart 5. On fra-node3: gluster system:: execute gsec_create gluster volume geo-replication fra-volume geoaccount@chi-node3::chi-volume create push-pem gluster volume geo-replication fra-volume geoaccount@chi-node3::chi-volume config use_meta_volume true gluster volume geo-replication fra-volume geoaccount@chi-node3::chi-volume start 6. On chi-node3: /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh geoaccount fra-volume chi-volume Volume an geo-replication status: $ vagrant ssh fra-node1 -c 'sudo gluster volume status fra-volume' ; \ vagrant ssh chi-node1 -c 'sudo gluster volume status chi-volume'; \ vagrant ssh fra-node1 -c 'sudo gluster volume geo-replication status' Status of volume: fra-volume Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick fra-node1:/data1/glusterfs 49152 0 Y 4499 Brick fra-node2:/data1/glusterfs 49152 0 Y 4434 Brick fra-node3:/data1/glusterfs 49152 0 Y 3753 Self-heal Daemon on localhost N/A N/A Y 5091 Self-heal Daemon on fra-node3.vagrant.finan cial.com N/A N/A Y 4623 Self-heal Daemon on fra-node2 N/A N/A Y 5024 Task Status of Volume fra-volume ------------------------------------------------------------------------------ There are no active volume tasks Connection to 127.0.0.1 closed. Status of volume: chi-volume Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick chi-node1:/data1/glusterfs 49152 0 Y 4498 Brick chi-node2:/data1/glusterfs 49152 0 Y 4435 Brick chi-node3:/data1/glusterfs 49152 0 Y 3750 Self-heal Daemon on localhost N/A N/A Y 5756 Self-heal Daemon on chi-node3.vagrant.finan cial.com N/A N/A Y 5900 Self-heal Daemon on chi-node2 N/A N/A Y 5667 Task Status of Volume chi-volume ------------------------------------------------------------------------------ There are no active volume tasks Connection to 127.0.0.1 closed. MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ----------------------------------------------------------------------------------------------------------------------------------------------------------- fra-node1 fra-volume /data1/glusterfs geoaccount ssh://geoaccount@chi-node3::chi-volume N/A Faulty N/A N/A fra-node3 fra-volume /data1/glusterfs geoaccount ssh://geoaccount@chi-node3::chi-volume N/A Faulty N/A N/A fra-node2 fra-volume /data1/glusterfs geoaccount ssh://geoaccount@chi-node3::chi-volume N/A Faulty N/A N/A How can I convince glusterd to create that template with correct permissions? Cheers, Stefan _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
