[2019-03-22 09:48:21.546308] E [syncdutils(worker /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):305:log_raise_exception] <top>: connection to peer is broken
[2019-03-22 09:48:21.546662] E [syncdutils(worker /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):309:log_raise_exception] <top>: getting "No such file or directory"errors is most likely due to MISCONFIGURATION, please remove all the public keys added by geo-replication from authorized_keys file in slave nodes and run Geo-replication create command again.
[2019-03-22 09:48:21.546736] E [syncdutils(worker /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):316:log_raise_exception] <top>: If `gsec_create container` was used, then run `gluster volume geo-replication <MASTERVOL> [<SLAVEUSER>@]<SLAVEHOST>::<SLAVEVOL> config remote-gsyncd <GSYNCD_PATH> (Example GSYNCD_PATH: `/usr/libexec/glusterfs/gsyncd`)
[2019-03-22 09:48:21.546858] E [syncdutils(worker /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):801:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-OaPGc3/c784230c9648efa4d529975bd779c551.sock azureuser@172.16.201.35 /nonexistent/gsyncd slave vol_041afbc53746053368a1840607636e97 azureuser@172.16.201.35::vol_a5aee81a873c043c99a938adcb5b5781 --master-node 172.16.189.4 --master-node-id dd4efc35-4b86-4901-9c00-483032614c35 --master-brick /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick --local-node 172.16.201.35 --local-node-id 7eb0a2b6-c4d6-41b1-a346-0638dbf8d779 --slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin error=127
[2019-03-22 09:48:21.546977] E [syncdutils(worker /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):805:logerr] Popen: ssh> bash: /nonexistent/gsyncd: No such file or directory
[2019-03-22 09:48:21.565583] I [repce(agent /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):80:service_loop] RepceServer: terminating on reaching EOF.
[2019-03-22 09:48:21.565745] I [monitor(monitor):266:monitor] Monitor: worker died before establishing connection brick=/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick
[2019-03-22 09:48:21.579195] I [gsyncdstatus(monitor):245:set_worker_status] GeorepStatus: Worker Status Change status=Faulty
Hi Maurya,
Looks like hook script is failed to set permissions for azureuser on
"/var/log/glusterfs".
You can assign permission manually for directory then it will work.
-Sunny
On Fri, Mar 22, 2019 at 2:07 PM Maurya M <mauryam@xxxxxxxxx> wrote:
>
> hi Sunny,
> Passwordless ssh to :
>
> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 azureuser@172.16.201.35
>
> is login, but when the whole command is run getting permission issues again::
>
> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 azureuser@172.16.201.35 gluster --xml --remote-host=localhost volume info vol_a5aee81a873c043c99a938adcb5b5781 -v
> ERROR: failed to create logfile "/var/log/glusterfs/cli.log" (Permission denied)
> ERROR: failed to open logfile /var/log/glusterfs/cli.log
>
> any idea here ?
>
> thanks,
> Maurya
>
>
> On Thu, Mar 21, 2019 at 2:43 PM Maurya M <mauryam@xxxxxxxxx> wrote:
>>
>> hi Sunny,
>> i did use the [1] link for the setup, when i encountered this error during ssh-copy-id : (so setup the passwordless ssh, by manually copied the private/ public keys to all the nodes , both master & slave)
>>
>> [root@k8s-agentpool1-24779565-1 ~]# ssh-copy-id geouser@xxxxxxxxxx.x
>> /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
>> The authenticity of host ' xxx.xx.xxx.x ( xxx.xx.xxx.x )' can't be established.
>> ECDSA key fingerprint is SHA256:B2rNaocIcPjRga13oTnopbJ5KjI/7l5fMANXc+KhA9s.
>> ECDSA key fingerprint is MD5:1b:70:f9:7a:bf:35:33:47:0c:f2:c1:cd:21:e2:d3:75.
>> Are you sure you want to continue connecting (yes/no)? yes
>> /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
>> /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
>> Permission denied (publickey).
>>
>> To start afresh what all needs to teardown / delete, do we have any script for it ? where all the pem keys do i need to delete?
>>
>> thanks,
>> Maurya
>>
>> On Thu, Mar 21, 2019 at 2:12 PM Sunny Kumar <sunkumar@xxxxxxxxxx> wrote:
>>>
>>> Hey you can start a fresh I think you are not following proper setup steps.
>>>
>>> Please follow these steps [1] to create geo-rep session, you can
>>> delete the old one and do a fresh start. Or alternative you can use
>>> this tool[2] to setup geo-rep.
>>>
>>>
>>> [1]. https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>>> [2]. http://aravindavk.in/blog/gluster-georep-tools/
>>>
>>>
>>> /Sunny
>>>
>>> On Thu, Mar 21, 2019 at 11:28 AM Maurya M <mauryam@xxxxxxxxx> wrote:
>>> >
>>> > Hi Sunil,
>>> > I did run the on the slave node :
>>> > /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
>>> > getting this message "/home/azureuser/common_secret.pem.pub not present. Please run geo-replication command on master with push-pem option to generate the file"
>>> >
>>> > So went back and created the session again, no change, so manually copied the common_secret.pem.pub to /home/azureuser/ but still the set_geo_rep_pem_keys.sh is looking the pem file in different name : COMMON_SECRET_PEM_PUB=${master_vol}_${slave_vol}_common_secret.pem.pub , change the name of pem , ran the command again :
>>> >
>>> > /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
>>> > Successfully copied file.
>>> > Command executed successfully.
>>> >
>>> >
>>> > - went back and created the session , start the geo-replication , still seeing the same error in logs. Any ideas ?
>>> >
>>> > thanks,
>>> > Maurya
>>> >
>>> >
>>> >
>>> > On Wed, Mar 20, 2019 at 11:07 PM Sunny Kumar <sunkumar@xxxxxxxxxx> wrote:
>>> >>
>>> >> Hi Maurya,
>>> >>
>>> >> I guess you missed last trick to distribute keys in slave node. I see
>>> >> this is non-root geo-rep setup so please try this:
>>> >>
>>> >>
>>> >> Run the following command as root in any one of Slave node.
>>> >>
>>> >> /usr/local/libexec/glusterfs/set_geo_rep_pem_keys.sh <slave_user>
>>> >> <master_volume> <slave_volume>
>>> >>
>>> >> - Sunny
>>> >>
>>> >> On Wed, Mar 20, 2019 at 10:47 PM Maurya M <mauryam@xxxxxxxxx> wrote:
>>> >> >
>>> >> > Hi all,
>>> >> > Have setup a 3 master nodes - 3 slave nodes (gluster 4.1) for geo-replication, but once have the geo-replication configure the status is always on "Created',
>>> >> > even after have force start the session.
>>> >> >
>>> >> > On close inspect of the logs on the master node seeing this error:
>>> >> >
>>> >> > "E [syncdutils(monitor):801:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 azureuser@xxxxxxxxxxxxxxx. gluster --xml --remote-host=localhost volume info vol_a5ae34341a873c043c99a938adcb5b5781 error=255"
>>> >> >
>>> >> > Any ideas what is issue?
>>> >> >
>>> >> > thanks,
>>> >> > Maurya
>>> >> >
>>> >> > _______________________________________________
>>> >> > Gluster-users mailing list
>>> >> > Gluster-users@xxxxxxxxxxx
>>> >> > https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
