Hi, I need help of solving geo-replication problem that is reproduced each time when geo-replication is deleted, some changes are made to the former master volume and the geo-replication is re-created to the same volume without deleting existing files from the former slave volume. Not all changes are synced to the slave volume after geo-replication is recreated. The list of failed changes consists of the next file and directory operations on the former master volume: - file deletion using command "rm"; - renaming non empty directory by using command "mv"; - file renaming by using command "mv"; - move file between directories by using command "mv"; - delete non empty directory, re-create under the same name with new files; After the geo-replication is re-created without deleting the slave volume files, some new changes on master volume like "add" or "modify" files are synced to the slave volume except changes listed above. I'm attaching glusterfs logs from master and slave nodes that are covering the test interval. The attached text file " geo_recreate_problem.txt" describes all steps that were done during testing to reproduce the problem. Best regards, Viktor Nosov
Master node: [root@SC-182 log]# glusterfsd --version glusterfs 3.7.16 built on Nov 1 2016 10:41:53 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Slave node: [root@SC-183 log]# glusterfsd --version glusterfs 3.7.16 built on Nov 1 2016 10:41:53 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Master volume: [root@SC-182 log]# gluster volume info nas-182-master-volume-0001 Volume Name: nas-182-master-volume-0001 Type: Distribute Volume ID: d37c2578-d509-4bc0-b997-f9cf95e9be6f Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.10.60.182:/exports/nas-182-brick-0001/nas-182-master-volume-0001 Options Reconfigured: server.allow-insecure: on performance.quick-read: off performance.stat-prefetch: off nfs.disable: on nfs.addr-namelookup: off performance.readdir-ahead: on cluster.enable-shared-storage: disable snap-activate-on-create: enable ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Master volume is mounted and has files: [root@SC-182 log]# mount /dev/sdc4 on / type ext3 (rw,acl) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sdc2 on /boot type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) /dev/sdg on /exports/nas-182-brick-0001 type xfs (rw,noatime,nodiratime,barrier,nouuid,inode64,logbufs=8,logbsize=256k) 127.0.0.1:/nas-182-master-volume-0001 on /samba/nas-182-master-volume-0001 type fuse.glusterfs (rw,allow_other,max_read=131072) [root@SC-182 log]# ls -l -R /samba/nas-182-master-volume-0001/cifs_share/ /samba/nas-182-master-volume-0001/cifs_share/: total 28 drwxr-xr-x 2 root root 4096 Dec 20 10:43 del_file_dir drwxr-xr-x 2 root root 4096 Dec 20 11:15 dir0 drwxr-xr-x 2 root root 4096 Dec 20 11:10 mv_dir_before drwxr-xr-x 2 root root 4096 Dec 20 11:03 mv_file_dir drwxr-xr-x 2 root root 4096 Dec 20 11:05 mv_file_from_dir drwxr-xr-x 2 root root 4096 Dec 20 10:33 mv_file_to_dir drwxr-xr-x 2 root root 4096 Dec 20 11:07 recreate_dir /samba/nas-182-master-volume-0001/cifs_share/del_file_dir: total 1 -rw-r--r-- 1 root root 69 Dec 20 10:43 file_to_delete -rw-r--r-- 1 root root 69 Dec 20 10:43 test.file /samba/nas-182-master-volume-0001/cifs_share/dir0: total 1 -rw-r--r-- 1 root root 69 Dec 20 10:30 test.file /samba/nas-182-master-volume-0001/cifs_share/mv_dir_before: total 2 -rw-r--r-- 1 root root 69 Dec 20 11:06 file_1 -rw-r--r-- 1 root root 69 Dec 20 11:06 file_2 -rw-r--r-- 1 root root 69 Dec 20 11:06 file_3 /samba/nas-182-master-volume-0001/cifs_share/mv_file_dir: total 1 -rw-r--r-- 1 root root 69 Dec 20 11:03 file_to_rename -rw-r--r-- 1 root root 69 Dec 20 11:01 test.file /samba/nas-182-master-volume-0001/cifs_share/mv_file_from_dir: total 1 -rw-r--r-- 1 root root 69 Dec 20 11:05 file_to_move /samba/nas-182-master-volume-0001/cifs_share/mv_file_to_dir: total 0 /samba/nas-182-master-volume-0001/cifs_share/recreate_dir: total 2 -rw-r--r-- 1 root root 69 Dec 20 11:07 file_a -rw-r--r-- 1 root root 69 Dec 20 11:07 file_b -rw-r--r-- 1 root root 69 Dec 20 11:07 file_c ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Slave volume: [root@SC-183 log]# gluster volume info nas-183-slave-volume-0001 Volume Name: nas-183-slave-volume-0001 Type: Distribute Volume ID: 328b3f19-4085-4715-8632-2bc6d46e8bd5 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.10.60.183:/exports/nas-183-brick-0001/nas-183-slave-volume-0001 Options Reconfigured: server.allow-insecure: on performance.quick-read: off performance.stat-prefetch: off nfs.disable: on nfs.addr-namelookup: off performance.readdir-ahead: on ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ No files on the slave volume: [root@SC-183 log]# mount /dev/sdc4 on / type ext3 (rw,acl) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sdc2 on /boot type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) /dev/sdg on /exports/nas-183-brick-0001 type xfs (rw,noatime,nodiratime,barrier,nouuid,inode64,logbufs=8,logbsize=256k) [root@SC-183 log]# ls -l -R /exports/nas-183-brick-0001 /exports/nas-183-brick-0001: total 0 drwxrwxrwx 4 root root 39 Dec 20 11:24 nas-183-slave-volume-0001 /exports/nas-183-brick-0001/nas-183-slave-volume-0001: total 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Create geo-replication: [root@SC-182 log]# date Tue Dec 20 11:37:48 PST 2016 7123 12/20/2016 11:38:31.854648608 1482262711 command: /usr/sbin/gluster system:: execute gsec_create 7123 12/20/2016 11:38:32.199841216 1482262712 status=0 /usr/sbin/gluster system:: execute gsec_create 7257 12/20/2016 11:38:44.105259744 1482262724 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 create push-pem 2>&1 7257 12/20/2016 11:39:00.467536723 1482262740 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 create push-pem 2>&1 7947 12/20/2016 11:39:06.275873228 1482262746 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 start 2>&1 7947 12/20/2016 11:39:08.015591501 1482262748 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 start 2>&1 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Verify that files are copyed to the slave volume: 10390 12/20/2016 11:42:04.099286832 1482262924 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 config checkpoint now 2>&1 10390 12/20/2016 11:42:05.155865944 1482262925 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 config checkpoint now 2>&1 10468 12/20/2016 11:42:05.171436491 1482262925 command: /bin/touch -a /samba/nas-182-master-volume-0001/ 10468 12/20/2016 11:42:05.175399020 1482262925 status=0 /bin/touch -a /samba/nas-182-master-volume-0001/ [root@SC-182 log]# /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.10.60.182 nas-182-master-volume-0001 /exports/nas-182-brick-0001/nas-182-master-volume-0001 nasgorep nasgorep@10.10.60.183::nas-183-slave-volume-0001 10.10.60.183 Active Changelog Crawl 2016-12-20 11:42:14 0 0 0 0 2016-12-20 11:42:04 Yes 2016-12-20 11:42:18 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Verify that files are on slave volume: Slave node: [root@SC-183 log]# date Tue Dec 20 11:44:01 PST 2016 [root@SC-183 log]# [root@SC-183 log]# ls -l -R /e etc/ exports/ [root@SC-183 log]# ls -l -R /exports/nas-183-brick-0001/ /exports/nas-183-brick-0001/: total 0 drwxrwxrwx 5 root root 56 Dec 20 10:25 nas-183-slave-volume-0001 /exports/nas-183-brick-0001/nas-183-slave-volume-0001: total 0 drwxrwxrwx 9 root root 137 Dec 20 11:14 cifs_share /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share: total 0 drwxr-xr-x 2 root root 43 Dec 20 10:43 del_file_dir drwxr-xr-x 2 root root 22 Dec 20 11:15 dir0 drwxr-xr-x 2 root root 45 Dec 20 11:10 mv_dir_before drwxr-xr-x 2 root root 43 Dec 20 11:03 mv_file_dir drwxr-xr-x 2 root root 25 Dec 20 11:05 mv_file_from_dir drwxr-xr-x 2 root root 6 Dec 20 10:33 mv_file_to_dir drwxr-xr-x 2 root root 45 Dec 20 11:07 recreate_dir /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/del_file_dir: total 8 -rw-r--r-- 2 root root 69 Dec 20 10:43 file_to_delete -rw-r--r-- 2 root root 69 Dec 20 10:43 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/dir0: total 4 -rw-r--r-- 2 root root 69 Dec 20 10:30 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_dir_before: total 12 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_1 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_2 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_3 /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_dir: total 8 -rw-r--r-- 2 root root 69 Dec 20 11:03 file_to_rename -rw-r--r-- 2 root root 69 Dec 20 11:01 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_from_dir: total 4 -rw-r--r-- 2 root root 69 Dec 20 11:05 file_to_move /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_to_dir: total 0 /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/recreate_dir: total 12 -rw-r--r-- 2 root root 69 Dec 20 11:07 file_a -rw-r--r-- 2 root root 69 Dec 20 11:07 file_b -rw-r--r-- 2 root root 69 Dec 20 11:07 file_c ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Delete geo-replication, do not use 'reset-sync-time'. Master node: [root@SC-182 log]# date Tue Dec 20 11:47:44 PST 2016 14306 12/20/2016 11:48:13.725156425 1482263293 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 stop 2>&1 14306 12/20/2016 11:48:15.966155330 1482263295 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 stop 2>&1 14927 12/20/2016 11:48:47.399252316 1482263327 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 delete 2>&1 14927 12/20/2016 11:48:48.975923450 1482263328 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 delete 2>&1 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Make changes on the master volume: [root@SC-182 log]# date Tue Dec 20 11:52:15 PST 2016 ========= Delete file from the directory ==================== [root@SC-182 log]# rm /samba/nas-182-master-volume-0001/cifs_share/del_file_dir/file_to_delete rm: remove regular file `/samba/nas-182-master-volume-0001/cifs_share/del_file_dir/file_to_delete'? y ========= Rename file at the directory ==================== [root@SC-182 log]# mv /samba/nas-182-master-volume-0001/cifs_share/mv_file_dir/file_to_rename /samba/nas-182-master-volume-0001/cifs_share/mv_file_dir/new_file_name ========= Move file between directories ==================== [root@SC-182 log]# mv /samba/nas-182-master-volume-0001/cifs_share/mv_file_from_dir/file_to_move /samba/nas-182-master-volume-0001/cifs_share/mv_file_to_dir/ ========= Rename directory ==================== [root@SC-182 log]# mv /samba/nas-182-master-volume-0001/cifs_share/mv_dir_before /samba/nas-182-master-volume-0001/cifs_share/mv_dir_after ========= Remove directory and recreate with new files ==================== [root@SC-182 log]# rm -rf /samba/nas-182-master-volume-0001/cifs_share/recreate_dir [root@SC-182 log]# mkdir /samba/nas-182-master-volume-0001/cifs_share/recreate_dir [root@SC-182 log]# cp test_after.file /samba/nas-182-master-volume-0001/cifs_share/recreate_dir/file_e [root@SC-182 log]# cp test_after.file /samba/nas-182-master-volume-0001/cifs_share/recreate_dir/file_d [root@SC-182 log]# cp test_after.file /samba/nas-182-master-volume-0001/cifs_share/recreate_dir/file_f ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Have new contents of the master volume: [root@SC-182 log]# ls -l -R /samba/nas-182-master-volume-0001/cifs_share/ /samba/nas-182-master-volume-0001/cifs_share/: total 28 drwxr-xr-x 2 root root 4096 Dec 20 11:53 del_file_dir drwxr-xr-x 2 root root 4096 Dec 20 11:15 dir0 drwxr-xr-x 2 root root 4096 Dec 20 11:10 mv_dir_after drwxr-xr-x 2 root root 4096 Dec 20 11:55 mv_file_dir drwxr-xr-x 2 root root 4096 Dec 20 11:56 mv_file_from_dir drwxr-xr-x 2 root root 4096 Dec 20 11:56 mv_file_to_dir drwxr-xr-x 2 root root 4096 Dec 20 12:02 recreate_dir /samba/nas-182-master-volume-0001/cifs_share/del_file_dir: total 1 -rw-r--r-- 1 root root 69 Dec 20 10:43 test.file /samba/nas-182-master-volume-0001/cifs_share/dir0: total 1 -rw-r--r-- 1 root root 69 Dec 20 10:30 test.file /samba/nas-182-master-volume-0001/cifs_share/mv_dir_after: total 2 -rw-r--r-- 1 root root 69 Dec 20 11:06 file_1 -rw-r--r-- 1 root root 69 Dec 20 11:06 file_2 -rw-r--r-- 1 root root 69 Dec 20 11:06 file_3 /samba/nas-182-master-volume-0001/cifs_share/mv_file_dir: total 1 -rw-r--r-- 1 root root 69 Dec 20 11:03 new_file_name -rw-r--r-- 1 root root 69 Dec 20 11:01 test.file /samba/nas-182-master-volume-0001/cifs_share/mv_file_from_dir: total 0 /samba/nas-182-master-volume-0001/cifs_share/mv_file_to_dir: total 1 -rw-r--r-- 1 root root 69 Dec 20 11:05 file_to_move /samba/nas-182-master-volume-0001/cifs_share/recreate_dir: total 2 -rw-r--r-- 1 root root 109 Dec 20 12:02 file_d -rw-r--r-- 1 root root 109 Dec 20 12:02 file_e -rw-r--r-- 1 root root 109 Dec 20 12:02 file_f [root@SC-182 log]# date Tue Dec 20 12:03:48 PST 2016 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Current contents of the slave volume: Slave node: [root@SC-183 log]# date Tue Dec 20 12:04:18 PST 2016 [root@SC-183 log]# ls -l -R /exports/nas-183-brick-0001/ /exports/nas-183-brick-0001/: total 0 drwxrwxrwx 5 root root 56 Dec 20 10:25 nas-183-slave-volume-0001 /exports/nas-183-brick-0001/nas-183-slave-volume-0001: total 0 drwxrwxrwx 9 root root 137 Dec 20 11:14 cifs_share /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share: total 0 drwxr-xr-x 2 root root 43 Dec 20 10:43 del_file_dir drwxr-xr-x 2 root root 22 Dec 20 11:15 dir0 drwxr-xr-x 2 root root 45 Dec 20 11:10 mv_dir_before drwxr-xr-x 2 root root 43 Dec 20 11:03 mv_file_dir drwxr-xr-x 2 root root 25 Dec 20 11:05 mv_file_from_dir drwxr-xr-x 2 root root 6 Dec 20 10:33 mv_file_to_dir drwxr-xr-x 2 root root 45 Dec 20 11:07 recreate_dir /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/del_file_dir: total 8 -rw-r--r-- 2 root root 69 Dec 20 10:43 file_to_delete -rw-r--r-- 2 root root 69 Dec 20 10:43 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/dir0: total 4 -rw-r--r-- 2 root root 69 Dec 20 10:30 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_dir_before: total 12 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_1 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_2 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_3 /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_dir: total 8 -rw-r--r-- 2 root root 69 Dec 20 11:03 file_to_rename -rw-r--r-- 2 root root 69 Dec 20 11:01 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_from_dir: total 4 -rw-r--r-- 2 root root 69 Dec 20 11:05 file_to_move /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_to_dir: total 0 /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/recreate_dir: total 12 -rw-r--r-- 2 root root 69 Dec 20 11:07 file_a -rw-r--r-- 2 root root 69 Dec 20 11:07 file_b -rw-r--r-- 2 root root 69 Dec 20 11:07 file_c ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Re-create geo-replication, do not delete files on the slave volume: Master node: [root@SC-182 log]# date Tue Dec 20 12:06:11 PST 2016 24191 12/20/2016 12:06:57.721883866 1482264417 command: /usr/sbin/gluster system:: execute gsec_create 24191 12/20/2016 12:06:58.193680431 1482264418 status=0 /usr/sbin/gluster system:: execute gsec_create 24301 12/20/2016 12:07:10.157581462 1482264430 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 create push-pem 2>&1 24301 12/20/2016 12:07:20.910094106 1482264440 status=1 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 create push-pem 2>&1 24732 12/20/2016 12:07:38.889286715 1482264458 command: /usr/sbin/gluster system:: execute gsec_create 24732 12/20/2016 12:07:39.076429219 1482264459 status=0 /usr/sbin/gluster system:: execute gsec_create 24766 12/20/2016 12:07:39.103350042 1482264459 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 create push-pem force 2>&1 24766 12/20/2016 12:07:55.515499228 1482264475 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 create push-pem force 2>&1 25462 12/20/2016 12:08:01.324579165 1482264481 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 start force 2>&1 25462 12/20/2016 12:08:03.001213345 1482264483 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 start force 2>&1 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Geo-replication status: [root@SC-182 log]# /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 10.10.60.182 nas-182-master-volume-0001 /exports/nas-182-brick-0001/nas-182-master-volume-0001 nasgorep nasgorep@10.10.60.183::nas-183-slave-volume-0001 10.10.60.183 Active Changelog Crawl 2016-12-20 12:08:09 0 0 0 9 N/A N/A N/A ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Verfy that geo-replication does not have pending operations: 27873 12/20/2016 12:11:04.528375312 1482264664 command: /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 config checkpoint now 2>&1 27873 12/20/2016 12:11:05.585733739 1482264665 status=0 /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 config checkpoint now 2>&1 27951 12/20/2016 12:11:05.601266338 1482264665 command: /bin/touch -a /samba/nas-182-master-volume-0001/ 27951 12/20/2016 12:11:05.605293641 1482264665 status=0 /bin/touch -a /samba/nas-182-master-volume-0001/ [root@SC-182 log]# /usr/sbin/gluster volume geo-replication nas-182-master-volume-0001 nasgorep@10.10.60.183::nas-183-slave-volume-0001 status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.10.60.182 nas-182-master-volume-0001 /exports/nas-182-brick-0001/nas-182-master-volume-0001 nasgorep nasgorep@10.10.60.183::nas-183-slave-volume-0001 10.10.60.183 Active Changelog Crawl 2016-12-20 12:11:09 0 0 0 9 2016-12-20 12:11:04 Yes 2016-12-20 12:11:11 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Check contents of the slave volume: Slave node: [root@SC-183 log]# date Tue Dec 20 12:12:54 PST 2016 [root@SC-183 log]# ls -l -R /exports/nas-183-brick-0001/ /exports/nas-183-brick-0001/: total 0 drwxrwxrwx 5 root root 56 Dec 20 10:25 nas-183-slave-volume-0001 /exports/nas-183-brick-0001/nas-183-slave-volume-0001: total 0 drwxrwxrwx 9 root root 137 Dec 20 12:01 cifs_share /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share: total 0 drwxr-xr-x 2 root root 43 Dec 20 11:53 del_file_dir drwxr-xr-x 2 root root 22 Dec 20 11:15 dir0 drwxr-xr-x 2 root root 45 Dec 20 11:10 mv_dir_before drwxr-xr-x 2 root root 43 Dec 20 11:55 mv_file_dir drwxr-xr-x 2 root root 25 Dec 20 11:56 mv_file_from_dir drwxr-xr-x 2 root root 6 Dec 20 11:56 mv_file_to_dir drwxr-xr-x 2 root root 45 Dec 20 11:07 recreate_dir /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/del_file_dir: total 8 -rw-r--r-- 2 root root 69 Dec 20 10:43 file_to_delete ======> FAILURE, file is still on the slave! -rw-r--r-- 2 root root 69 Dec 20 10:43 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/dir0: total 4 -rw-r--r-- 2 root root 69 Dec 20 10:30 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_dir_before: ======> FAILURE, dir is still on the slave, new dir is absent! total 12 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_1 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_2 -rw-r--r-- 2 root root 69 Dec 20 11:06 file_3 /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_dir: total 8 -rw-r--r-- 2 root root 69 Dec 20 11:03 file_to_rename ======> FAILURE, file has the old name! -rw-r--r-- 2 root root 69 Dec 20 11:01 test.file /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_from_dir: total 4 -rw-r--r-- 2 root root 69 Dec 20 11:05 file_to_move ======> FAILURE, file is still in old dir! /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/mv_file_to_dir: total 0 /exports/nas-183-brick-0001/nas-183-slave-volume-0001/cifs_share/recreate_dir: ======> FAILURE, dir has old contents! total 12 -rw-r--r-- 2 root root 69 Dec 20 11:07 file_a -rw-r--r-- 2 root root 69 Dec 20 11:07 file_b -rw-r--r-- 2 root root 69 Dec 20 11:07 file_c
Attachment:
node_182_logs.tar
Description: Unix tar archive
Attachment:
node_183_logs.tar
Description: Unix tar archive
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
