the port 2503 is listening on all gluster nodes and i can
successfully login to the participating slaves...
[ 09:55:15 ] - root@gluster-ger-ber-09 ~/.ssh $ssh -p 2503
gluster-wien-05 hostname
gluster-wien-05
[ 09:55:19 ] - root@gluster-ger-ber-09 ~/.ssh $
the authorized_keys on all slaves contain corresponding lines for
the tar command and gsyncd for each master node.
the gsyncd.conf on all master nodes are prepared to use port 2503
for the ssh_command and the ssh_command_tar.
privilege-separation is used on all nodes...:
UsePrivilegeSeparation yes
i have to examine if it possibly has changed by ubuntu upgrade,
openssh package was upgraded in 14.04...in 12.04 with 3.4.7
geo-replication worked fine on port 2503.
i just changed UsePrivilegeSeparation to 'no' but for the first 10
min nothing changed...
because the slave volume already contain about 99% of the master
data by the former 3.4.7 geo-replication i assumed changelog could
be the right setting...
does anybody have further hints...?
best regards
dietmar
On 11.11.2015 02:38, Wade Fitzpatrick
wrote:
Your ssh commands connect to port 2503 - is that port listening on
the slaves?
Does it use privilege-separation?
On 11/11/2015 3:20 AM, Dietmar Putz
wrote:
Hi
all,
i need some help with a geo-replication issue...
recently i upgraded two 6-node distributed-replicated gluster
from ubuntu 12.04.5 lts to 14.04.3 lts resp. glusterfs 3.4.7 to
3.5.6
since then the geo-replication does not start syncing but
remains as shown in the 'status detail' output below for about
48h.
I followed the hints for upgrade with an existing
geo-replication :
http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5
the master_gfid_file.txt was created and applied to the slave
volume. geo-replication was started with 'force' option.
in the gluster.log on the slave i can find thousands of lines
with messages like :
".../.gfid/1abb953b-aa9d-4fa3-9a72-415204057572 => -1
(Operation not permitted)"
and no files are synced.
I'm not sure whats going on and since there are about 40TByte of
data already replicated by the old 3.4.7 setup I have some fear
to try around...
so i got some questions...maybe somebody can give me some
hints...
1. as shown in the example below the trusted.gfid of the same
file differs in master and slave volume. as far as i understood
the upgrade-howto after applying the master_gfid_file.txt on the
slave they should be the same on master and slave...is that
right ?
2. as shown in the config below the change_detector is 'xsync'.
Somewhere i red that xsync is used for the initial replication
and is changing to 'change_log' later on when the entire sync is
done. should i try to modify the change_detector to
'change_log', does it make sense...?
any other idea which could help me to solve this problem....?
best regards
dietmar
[ 11:10:01 ] - root@gluster-ger-ber-09 ~ $glusterfs --version
glusterfs 3.5.6 built on Sep 16 2015 15:27:30
...
[ 11:11:37 ] - root@gluster-ger-ber-09 ~ $cat
/var/lib/glusterd/glusterd.info | grep operating-version
operating-version=30501
[ 10:55:35 ] - root@gluster-ger-ber-09 ~ $gluster volume
geo-replication ger-ber-01 ssh://gluster-wien-02::aut-wien-01
status detail
MASTER NODE MASTER VOL MASTER BRICK
SLAVE STATUS CHECKPOINT
STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES
PENDING DELETES PENDING FILES SKIPPED
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gluster-ger-ber-09 ger-ber-01 /gluster-export
gluster-wien-05-int::aut-wien-01 Active
N/A Hybrid Crawl 0 8191
0 0 0
gluster-ger-ber-11 ger-ber-01 /gluster-export ssh://gluster-wien-02::aut-wien-01
Not Started N/A N/A N/A
N/A N/A N/A N/A
gluster-ger-ber-10 ger-ber-01 /gluster-export ssh://gluster-wien-02::aut-wien-01
Not Started N/A N/A N/A
N/A N/A N/A N/A
gluster-ger-ber-12 ger-ber-01 /gluster-export ssh://gluster-wien-02::aut-wien-01
Not Started N/A N/A N/A
N/A N/A N/A N/A
gluster-ger-ber-07 ger-ber-01 /gluster-export ssh://gluster-wien-02::aut-wien-01
Not Started N/A N/A N/A
N/A N/A N/A N/A
gluster-ger-ber-08 ger-ber-01 /gluster-export
gluster-wien-04-int::aut-wien-01 Passive
N/A N/A 0 0
0 0 0
[ 10:55:48 ] - root@gluster-ger-ber-09 ~ $
[ 10:56:56 ] - root@gluster-ger-ber-09 ~ $gluster volume
geo-replication ger-ber-01 ssh://gluster-wien-02::aut-wien-01
config
special_sync_mode: partial
state_socket_unencoded:
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.socket
gluster_log_file:
/var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log
ssh_command: ssh -p 2503 -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem
ignore_deletes: true
change_detector: xsync
ssh_command_tar: ssh -p 2503 -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/tar_ssh.pem
working_dir:
/var/run/gluster/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01
remote_gsyncd: /nonexistent/gsyncd
log_file:
/var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.log
socketdir: /var/run
state_file:
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.status
state_detail_file:
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01-detail.status
session_owner: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
gluster_command_dir: /usr/sbin/
pid_file:
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.pid
georep_session_working_dir:
/var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/
gluster_params: aux-gfid-mount
volume_id: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
[ 11:10:01 ] - root@gluster-ger-ber-09 ~ $
[ 12:45:34 ] - root@gluster-wien-05
/var/log/glusterfs/geo-replication-slaves $tail -f
6a071cfa-b150-4f0b-b1ed-96ab5d4bd671\:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log
[2015-11-10 12:59:16.097932] W
[fuse-bridge.c:1942:fuse_create_cbk] 0-glusterfs-fuse: 54267:
/.gfid/1abb953b-aa9d-4fa3-9a72-415204057572 => -1 (Operation
not permitted)
[2015-11-10 12:59:16.098044] W [defaults.c:1381:default_release]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.6/xlator/mount/fuse.so(+0xfb4d)
[0x7fc9cd104b4d]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.6/xlator/mount/fuse.so(free_fuse_state+0x85)
[0x7fc9cd0fab95]
(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(fd_unref+0x10e)
[0x7fc9cf52ec9e]))) 0-fuse: xlator does not implement
release_cbk
...
grep 1abb953b-aa9d-4fa3-9a72-415204057572 master_gfid_file.txt
1abb953b-aa9d-4fa3-9a72-415204057572 1050/hyve/364/14158.mp4
putz@sdn-de-gate-01:~/central$ ./mycommand.sh -H
gluster-ger,gluster-wien -c "getfattr -m . -d -e hex
/gluster-export/1050/hyve/364/14158.mp4"
...
master volume :
-----------------------------------------------------
Host : gluster-ger-ber-09-int
# file: gluster-export/1050/hyve/364/14158.mp4
trusted.afr.ger-ber-01-client-6=0x000000000000000000000000
trusted.afr.ger-ber-01-client-7=0x000000000000000000000000
trusted.gfid=0x1abb953baa9d4fa39a72415204057572
trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f
-----------------------------------------------------
Host : gluster-ger-ber-10-int
# file: gluster-export/1050/hyve/364/14158.mp4
trusted.afr.ger-ber-01-client-6=0x000000000000000000000000
trusted.afr.ger-ber-01-client-7=0x000000000000000000000000
trusted.gfid=0x1abb953baa9d4fa39a72415204057572
trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f
...
slave volume :
Host : gluster-wien-04
# file: gluster-export/1050/hyve/364/14158.mp4
trusted.afr.aut-wien-01-client-2=0x000000000000000000000000
trusted.afr.aut-wien-01-client-3=0x000000000000000000000000
trusted.gfid=0x129ba62c3d214b34beb366fb1e2c8e4b
trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f
-----------------------------------------------------
Host : gluster-wien-05
# file: gluster-export/1050/hyve/364/14158.mp4
trusted.afr.aut-wien-01-client-2=0x000000000000000000000000
trusted.afr.aut-wien-01-client-3=0x000000000000000000000000
trusted.gfid=0x129ba62c3d214b34beb366fb1e2c8e4b
trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f
-----------------------------------------------------
...
putz@sdn-de-gate-01:~/central$
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
--
Dietmar Putz
3Q Medien GmbH
Wetzlarer Str. 86
D-14482 Potsdam
Telefax: +49 (0)331 / 2797 866 - 1
Telefon: +49 (0)331 / 2792 866 - 8
Mobile: +49 171 / 90 160 39
Mail: putz@xxxxxxxxxxxx
|
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
