Hello - I am having a problem with geo-replication on glusterv5 that I hope someone can help me with. I have a 7-server distribute cluster as the primary volume, and a 2 server distribute cluster as the secondary volume. Both are running the same version of gluster on CentOS 7: glusterfs-5.3-2.el7.x86_64 I was able to setup the replication keys, user, groups, etc and establish the session, but it goes faulty quickly after initializing. I ran into the missing libgfchangelog.so error and fixed with a symlink: [root@pcic-backup01 ~]# ln -s /usr/lib64/libgfchangelog.so.0 /usr/lib64/libgfchangelog.so [root@pcic-backup01 ~]# ls -lh /usr/lib64/libgfchangelog.so* lrwxrwxrwx. 1 root root 30 May 16 13:16 /usr/lib64/libgfchangelog.so -> /usr/lib64/libgfchangelog.so.0 lrwxrwxrwx. 1 root root 23 May 16 08:58 /usr/lib64/libgfchangelog.so.0 -> libgfchangelog.so.0.0.1 -rwxr-xr-x. 1 root root 62K Feb 25 04:02 /usr/lib64/libgfchangelog.so.0.0.1 But right now, when trying to start replication it goes faulty: [root@gluster01 ~]# gluster volume geo-replication storage geoaccount@10.0.231.81::pcic-backup start Starting geo-replication session between storage & geoaccount@10.0.231.81::pcic-backup has been successful [root@gluster01 ~]# gluster volume geo-replication status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.0.231.50 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Initializing... N/A N/A 10.0.231.54 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Initializing... N/A N/A 10.0.231.56 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Initializing... N/A N/A 10.0.231.52 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Initializing... N/A N/A 10.0.231.55 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Initializing... N/A N/A 10.0.231.51 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Initializing... N/A N/A 10.0.231.53 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Initializing... N/A N/A [root@gluster01 ~]# gluster volume geo-replication status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 10.0.231.50 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Faulty N/A N/A 10.0.231.54 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Faulty N/A N/A 10.0.231.56 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Faulty N/A N/A 10.0.231.55 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Faulty N/A N/A 10.0.231.53 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Faulty N/A N/A 10.0.231.51 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Faulty N/A N/A 10.0.231.52 storage /mnt/raid6-storage/storage geoaccount ssh://geoaccount@10.0.231.81::pcic-backup N/A Faulty N/A N/A [root@gluster01 ~]# gluster volume geo-replication storage geoaccount@10.0.231.81::pcic-backup stop Stopping geo-replication session between storage & geoaccount@10.0.231.81::pcic-backup has been successful And the /var/log/glusterfs/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.log log file contains the error: GLUSTER: Changelog register failed error=[Errno 21] Is a directory [root@gluster01 ~]# cat /var/log/glusterfs/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.log [2019-05-23 17:07:23.500781] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:23.629298] I [gsyncd(status):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:31.354005] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:31.483582] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:31.863888] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:31.994895] I [gsyncd(monitor):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:33.133888] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... [2019-05-23 17:07:33.134301] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/mnt/raid6-storage/storage slave_node=10.0.231.81 [2019-05-23 17:07:33.214462] I [gsyncd(agent /mnt/raid6-storage/storage):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:33.216737] I [changelogagent(agent /mnt/raid6-storage/storage):72:__init__] ChangelogAgent: Agent listining... [2019-05-23 17:07:33.228072] I [gsyncd(worker /mnt/raid6-storage/storage):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:33.247236] I [resource(worker /mnt/raid6-storage/storage):1366:connect_remote] SSH: Initializing SSH connection between master and slave... [2019-05-23 17:07:34.948796] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:35.73339] I [gsyncd(status):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:35.232405] I [resource(worker /mnt/raid6-storage/storage):1413:connect_remote] SSH: SSH connection between master and slave established. duration=1.9849 [2019-05-23 17:07:35.232748] I [resource(worker /mnt/raid6-storage/storage):1085:connect] GLUSTER: Mounting gluster volume locally... [2019-05-23 17:07:36.359250] I [resource(worker /mnt/raid6-storage/storage):1108:connect] GLUSTER: Mounted gluster volume duration=1.1262 [2019-05-23 17:07:36.359639] I [subcmds(worker /mnt/raid6-storage/storage):80:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor [2019-05-23 17:07:36.380975] E [repce(agent /mnt/raid6-storage/storage):122:worker] <top>: call failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 40, in register return Changes.cl_register(cl_brick, cl_dir, cl_log, cl_level, retries) File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 45, in cl_register cls.raise_changelog_err() File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 29, in raise_changelog_err raise ChangelogException(errn, os.strerror(errn)) ChangelogException: [Errno 21] Is a directory [2019-05-23 17:07:36.382556] E [repce(worker /mnt/raid6-storage/storage):214:__call__] RepceClient: call failed call=27412:140659114579776:1558631256.38 method=register error=ChangelogException [2019-05-23 17:07:36.382833] E [resource(worker /mnt/raid6-storage/storage):1266:service_loop] GLUSTER: Changelog register failed error=[Errno 21] Is a directory [2019-05-23 17:07:36.404313] I [repce(agent /mnt/raid6-storage/storage):97:service_loop] RepceServer: terminating on reaching EOF. [2019-05-23 17:07:37.361396] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase brick=/mnt/raid6-storage/storage [2019-05-23 17:07:37.370690] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2019-05-23 17:07:41.526408] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:41.643923] I [gsyncd(status):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:45.722193] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:45.817210] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:46.188499] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:46.258817] I [gsyncd(config-get):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:47.350276] I [gsyncd(monitor-status):308:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf [2019-05-23 17:07:47.364751] I [subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor Status Change status=Stopped I'm not really sure where to go from here... [root@gluster01 ~]# gluster volume geo-replication storage geoaccount@10.0.231.81::pcic-backup config | grep -i changelog change_detector:changelog changelog_archive_format:%Y%m changelog_batch_size:727040 changelog_log_file:/var/log/glusterfs/geo-replication/storage_10.0.231.81_pcic-backup/changes-${local_id}.log changelog_log_level:INFO Thanks, -Matthew _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users