Hi,
we need some help...we are running a 6 node distributed-replicated
master-volume which needs to be geo-replicated to another equal
configured volume.
recently both locations were updated from 3.4.7 to 3.5.6 to
3.6.7...running on ubuntu 14.04 lts.
since update to 3.5.6 the geo-replication remains in status 'hybrid
crawl' as shown below.
[ 16:31:46 ] - root@gluster-ger-ber-09 /home/admin $gluster volume
geo-replication ger-ber-01 gluster-wien-02::aut-wien-01 status detail
MASTER NODE MASTER VOL MASTER BRICK
SLAVE STATUS CHECKPOINT STATUS
CRAWL STATUS FILES SYNCD FILES PENDING BYTES PENDING
DELETES PENDING FILES SKIPPED
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gluster-ger-ber-09 ger-ber-01 /gluster-export
gluster-wien-05-int::aut-wien-01 Active N/A
Changelog Crawl 214645 4 0 2 11
gluster-ger-ber-12 ger-ber-01 /gluster-export
gluster-wien-06-int::aut-wien-01 Passive N/A
N/A 0 0 0 0 0
gluster-ger-ber-11 ger-ber-01 /gluster-export
gluster-wien-03-int::aut-wien-01 Active N/A Hybrid
Crawl 2831091 0 0 0 491
gluster-ger-ber-10 ger-ber-01 /gluster-export
gluster-wien-02-int::aut-wien-01 Passive N/A
N/A 0 0 0 0 0
gluster-ger-ber-07 ger-ber-01 /gluster-export
gluster-wien-07-int::aut-wien-01 Active N/A Hybrid
Crawl 2886662 0 0 0 474
gluster-ger-ber-08 ger-ber-01 /gluster-export
gluster-wien-04-int::aut-wien-01 Passive N/A
N/A 0 0 0 0 0
[ 16:31:51 ] - root@gluster-ger-ber-09 /home/admin $
on a slave we found a lot of repeating messages like :
root@gluster-wien-07 /var/log/glusterfs/geo-replication-slaves $tail -f
6a071cfa-b150-4f0b-b1ed-96ab5d4bd671\:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log
...
[2015-12-01 23:30:19.782137] W
[client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-2:
remote operation failed: File exists. Path:
<gfid:135991e9-5591-4217-b932-04cce065982f>/rfvg2cmFNJ8Xt9HG.png
[2015-12-01 23:30:19.782178] W
[client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-3:
remote operation failed: File exists. Path:
<gfid:135991e9-5591-4217-b932-04cce065982f>/rfvg2cmFNJ8Xt9HG.png
[2015-12-01 23:30:19.782573] W [fuse-bridge.c:1261:fuse_err_cbk]
0-glusterfs-fuse: 258115: SETXATTR()
/.gfid/135991e9-5591-4217-b932-04cce065982f => -1 (File exists)
...
all mentioned files have the same structure on the master volume...they
appear on two sub-volumes.
as you can see below, on one subvolume the mentioned file was created on
the 8th jun. on that day we had a rebalance on the master-volume.
I'm not really sure if this dht link file should have 0 bytes on this
subvolume ?
i already read this but i believe our situation is a bit more complicated...
https://www.gluster.org/pipermail/gluster-users/2015-February/020608.html
is it safe to delete those link files and the corresponding gfid on the
affected bricks ?
best regards
dietmar
tron@dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png master_gfid_file.txt
e95a43c2-4969-4662-8ea9-72e02e82df90
thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
tron@dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-ger-ber-*
ls-lisa-gluster.gluster-ger-ber-07-int.out:121247406422 612 -rw-rw-rw- 2
root root 619294 Okt 18 2014
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-ger-ber-08-int.out:121289252163 612 -rw-rw-rw- 2
root root 619294 Okt 18 2014
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-ger-ber-11-int.out:43282956834 4 ---------T 2
root root 619294 Jun 8 09:52
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-ger-ber-12-int.out:65271376731 4 ---------T 2
root root 619294 Jun 8 09:52
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
tron@dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-wien-0*
ls-lisa-gluster.gluster-wien-04.out:82283414551 612 -rw-rw-rw- 2 root
root 619294 Oct 18 2014
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-wien-05.out:82278502871 612 -rw-rw-rw- 2 root
root 619294 Oct 18 2014
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
tron@dp-server:~/gluster-9$
--
Dietmar Putz
3Q Medien GmbH
Wetzlarer Str. 86
D-14482 Potsdam
Telefax: +49 (0)331 / 2797 866 - 1
Telefon: +49 (0)331 / 2792 866 - 8
Mobile: +49 171 / 90 160 39
Mail: putz@xxxxxxxxxxxx
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
