Re: Unable to setup geo replication

["Tan, Jian Chern" <jian.chern.tan@xxxxxxxxx>]

Rsync on both the slave and master are rsync  version 3.1.3  protocol version 31, so both are up to date as far as I know.

Gluster version on both machines are glusterfs 5.10

OS on both machines are Fedora 29 Server Edition

 

From: Kotresh Hiremath Ravishankar <khiremat@xxxxxxxxxx>
Sent: Tuesday, November 26, 2019 3:04 PM
To: Tan, Jian Chern <jian.chern.tan@xxxxxxxxx>
Cc: gluster-users@xxxxxxxxxxx
Subject: Re: [Gluster-users] Unable to setup geo replication

 

The error code 14 related IPC where any of pipe/fork fails in rsync code.
Please upgrade the rsync if not done. Also check the rsync versions between master and slave to be same.

Which version of gluster are you using?
What's the host OS?

What's the rsync version ?

 

On Tue, Nov 26, 2019 at 11:34 AM Tan, Jian Chern <jian.chern.tan@xxxxxxxxx> wrote:

I’m new to GlusterFS and trying to setup geo-replication with a master volume being mirrored to a slave volume on another machine. However I just can’t seem to get it to work after starting the geo replication volume with the logs showing it failing rsync with error code 14. I can’t seem to find any info about this online so any help would be much appreciated.

 

[2019-11-26 05:46:31.24706] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change  status=Initializing...

[2019-11-26 05:46:31.24891] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker    brick=/data/glusterimagebrick/jfsotc22-gv0      slave_node=pgsotc11.png.intel.com

[2019-11-26 05:46:31.90935] I [gsyncd(agent /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file    path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf

[2019-11-26 05:46:31.92105] I [changelogagent(agent /data/glusterimagebrick/jfsotc22-gv0):72:__init__] ChangelogAgent: Agent listining...

[2019-11-26 05:46:31.93148] I [gsyncd(worker /data/glusterimagebrick/jfsotc22-gv0):308:main] <top>: Using session config file   path=/var/lib/glusterd/geo-replication/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/gsyncd.conf

[2019-11-26 05:46:31.102422] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1366:connect_remote] SSH: Initializing SSH connection between master and slave...

[2019-11-26 05:46:50.355233] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1413:connect_remote] SSH: SSH connection between master and slave established.    duration=19.2526

[2019-11-26 05:46:50.355583] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1085:connect] GLUSTER: Mounting gluster volume locally...

[2019-11-26 05:46:51.404998] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1108:connect] GLUSTER: Mounted gluster volume     duration=1.0492

[2019-11-26 05:46:51.405363] I [subcmds(worker /data/glusterimagebrick/jfsotc22-gv0):80:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor

[2019-11-26 05:46:53.431502] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1603:register] _GMaster: Working dir        path=/var/lib/misc/gluster/gsyncd/jfsotc22-gv0_pgsotc11.png.intel.com_pgsotc11-gv0/data-glusterimagebrick-jfsotc22-gv0

[2019-11-26 05:46:53.431846] I [resource(worker /data/glusterimagebrick/jfsotc22-gv0):1271:service_loop] GLUSTER: Register time time=1574747213

[2019-11-26 05:46:53.445589] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):281:set_active] GeorepStatus: Worker Status Change    status=Active

[2019-11-26 05:46:53.446184] I [gsyncdstatus(worker /data/glusterimagebrick/jfsotc22-gv0):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change        status=History Crawl

[2019-11-26 05:46:53.446367] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1517:crawl] _GMaster: starting history crawl        turns=1 stime=(1574669325, 0)   etime=1574747213        entry_stime=None

[2019-11-26 05:46:54.448994] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1546:crawl] _GMaster: slave's time  stime=(1574669325, 0)

[2019-11-26 05:46:54.928395] I [master(worker /data/glusterimagebrick/jfsotc22-gv0):1954:syncjob] Syncer: Sync Time Taken       job=1   num_files=1     return_code=14  duration=0.0162

[2019-11-26 05:46:54.928607] E [syncdutils(worker /data/glusterimagebrick/jfsotc22-gv0):809:errlog] Popen: command returned error       cmd=rsync -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-rgpu74f3/de0855b3336b4c3233934fcbeeb3674c.sock pgsotc11.png.intel.com:/proc/29549/cwd      error=14

[2019-11-26 05:46:54.935529] I [repce(agent /data/glusterimagebrick/jfsotc22-gv0):97:service_loop] RepceServer: terminating on reaching EOF.

[2019-11-26 05:46:55.410444] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase     brick=/data/glusterimagebrick/jfsotc22-gv0

[2019-11-26 05:46:55.412591] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty

[2019-11-26 05:47:05.631944] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...

….

 

Thanks!

Jian Chern

 

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

 

--

Thanks and Regards,

Kotresh H R

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users