It seems that it is no longer possible to stop, and delete a
GlusterFS volume, and then clean, and recreate it from the
component bricks, when the volume is "Distributed-Replicate".
The same steps used with a simple "Replicate" or "Distribute"
volume are successful with the correct data.
For the failing "Distributed-Replicate" case, the "rebalance"
always fails after the recreate. The "heal" command is
successful, showing no "info split-brain" files indicated, but
about half of the files are missing, and there are many
"(Possible split-brain)" warnings in the logfile.
The gluster "Distributed-Replicate" volume "recreate"
procedure works fine in GlusterFS versions 3.2.7, and 3.4.2,
but not in glusterfs 3.6.5, 3.6.7, or 3.7.6.
Perhaps the "recreate" procedure has changed, or I'm doing
something wrong that now matters in the newer GlusterFS
versions.
Details below. Any ideas how to make it work again?
Thanks.
~ Jeff Byers ~
# glusterd -V
glusterfs 3.7.6 built on Dec 14 2015 07:05:12
################################################
# Failing "Distributed-Replicate" recreate case.
################################################
# mountpoint /exports/test-dir/
/exports/test-dir/ is a mountpoint
# mount |grep test-dir
/dev/sdu on /exports/test-dir type xfs (rw,noatime,nodiratime,barrier,nouuid,inode64,logbufs=8,logbsize=256k)
# mkdir /exports/test-dir/test-brick-1a
# mkdir /exports/test-dir/test-brick-1b
# mkdir /exports/test-dir/test-brick-2a
# mkdir /exports/test-dir/test-brick-2b
# gluster volume create test-replica-dist replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-1a 10.10.60.169:/exports/test-dir/test-brick-2a 10.10.60.169:/exports/test-dir/test-brick-1b 10.10.60.169:/exports/test-dir/test-brick-2b force
volume create: test-replica-dist: success: please start the volume to access data
# gluster volume start test-replica-dist
volume start: test-replica-dist: success
# gluster volume info test-replica-dist
Volume Name: test-replica-dist
Type: Distributed-Replicate
Volume ID: c8de4e65-2304-4801-a244-6511f39fc0c9
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.10.60.169:/exports/test-dir/test-brick-1a
Brick2: 10.10.60.169:/exports/test-dir/test-brick-2a
Brick3: 10.10.60.169:/exports/test-dir/test-brick-1b
Brick4: 10.10.60.169:/exports/test-dir/test-brick-2b
Options Reconfigured:
snap-activate-on-create: enable
# mkdir /mnt/test-replica-dist
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica-dist /mnt/test-replica-dist/
# cp -rf /lib64/ /mnt/test-replica-dist/
# diff -r /lib64/ /mnt/test-replica-dist/lib64/
# umount /mnt/test-replica-dist
# gluster volume stop test-replica-dist
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test-replica-dist: success
# gluster volume delete test-replica-dist
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: test-replica-dist: success
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-1a
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1a
xattr clean-up in progress: /exports/test-dir/test-brick-1a
/exports/test-dir/test-brick-1a ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-1b
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1b
xattr clean-up in progress: /exports/test-dir/test-brick-1b
/exports/test-dir/test-brick-1b ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-2a
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2a
xattr clean-up in progress: /exports/test-dir/test-brick-2a
/exports/test-dir/test-brick-2a ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-2b
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2b
xattr clean-up in progress: /exports/test-dir/test-brick-2b
/exports/test-dir/test-brick-2b ready to be used as a glusterfs brick
# gluster volume create test-replica-dist replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-1a 10.10.60.169:/exports/test-dir/test-brick-2a 10.10.60.169:/exports/test-dir/test-brick-1b 10.10.60.169:/exports/test-dir/test-brick-2b force
volume create: test-replica-dist: success: please start the volume to access data
# gluster volume start test-replica-dist
volume start: test-replica-dist: success
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica-dist /mnt/test-replica-dist/
# diff -r /lib64/ /mnt/test-replica-dist/lib64/
Only in /lib64/device-mapper: libdevmapper-event-lvm2thin.so
Only in /lib64/multipath: libcheckcciss_tur.so
Only in /lib64/multipath: libcheckemc_clariion.so
Only in /lib64/multipath: libcheckhp_sw.so
Only in /lib64/multipath: libprioconst.so
Only in /lib64/multipath: libpriordac.so
Only in /lib64/multipath: libprioweighted.so
Only in /lib64/rtkaio: librtkaio-2.12.so
Only in /lib64/rtkaio: librt.so.1
Only in /lib64/xtables: libip6t_ah.so
Only in /lib64/xtables: libip6t_dst.so
Only in /lib64/xtables: libip6t_eui64.so
Only in /lib64/xtables: libip6t_frag.so
Only in /lib64/xtables: libip6t_HL.so
Only in /lib64/xtables: libip6t_icmp6.so
Only in /lib64/xtables: libip6t_LOG.so
Only in /lib64/xtables: libip6t_mh.so
Only in /lib64/xtables: libip6t_REJECT.so
Only in /lib64/xtables: libip6t_set.so
Only in /lib64/xtables: libipt_ah.so
Only in /lib64/xtables: libipt_ecn.so
Only in /lib64/xtables: libipt_ECN.so
Only in /lib64/xtables: libipt_icmp.so
Only in /lib64/xtables: libipt_MIRROR.so
Only in /lib64/xtables: libipt_realm.so
Only in /lib64/xtables: libipt_REDIRECT.so
Only in /lib64/xtables: libipt_REJECT.so
Only in /lib64/xtables: libipt_SAME.so
Only in /lib64/xtables: libipt_SET.so
Only in /lib64/xtables: libipt_SNAT.so
Only in /lib64/xtables: libipt_ttl.so
Only in /lib64/xtables: libipt_ULOG.so
Only in /lib64/xtables: libipt_unclean.so
Only in /lib64/xtables: libxt_CHECKSUM.so
Only in /lib64/xtables: libxt_cluster.so
Only in /lib64/xtables: libxt_connbytes.so
Only in /lib64/xtables: libxt_connlimit.so
Only in /lib64/xtables: libxt_CONNMARK.so
Only in /lib64/xtables: libxt_CONNSECMARK.so
Only in /lib64/xtables: libxt_conntrack.so
Only in /lib64/xtables: libxt_dccp.so
Only in /lib64/xtables: libxt_dscp.so
Only in /lib64/xtables: libxt_iprange.so
Only in /lib64/xtables: libxt_length.so
Only in /lib64/xtables: libxt_limit.so
Only in /lib64/xtables: libxt_mac.so
Only in /lib64/xtables: libxt_multiport.so
Only in /lib64/xtables: libxt_osf.so
Only in /lib64/xtables: libxt_physdev.so
Only in /lib64/xtables: libxt_pkttype.so
Only in /lib64/xtables: libxt_policy.so
Only in /lib64/xtables: libxt_quota.so
Only in /lib64/xtables: libxt_RATEEST.so
Only in /lib64/xtables: libxt_sctp.so
Only in /lib64/xtables: libxt_SECMARK.so
Only in /lib64/xtables: libxt_socket.so
Only in /lib64/xtables: libxt_statistic.so
Only in /lib64/xtables: libxt_string.so
Only in /lib64/xtables: libxt_tcpmss.so
Only in /lib64/xtables: libxt_TCPOPTSTRIP.so
Only in /lib64/xtables: libxt_time.so
Only in /lib64/xtables: libxt_TOS.so
Only in /lib64/xtables: libxt_TPROXY.so
Only in /lib64/xtables: libxt_TRACE.so
Only in /lib64/xtables: libxt_udp.so
# gluster volume rebalance test-replica-dist start
volume rebalance: test-replica-dist: success: Rebalance on test-replica-dist has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: ccf76757-c3df-4ae2-af2d-b82f8283d821
# gluster volume rebalance test-replica-dist status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes 249 3 0 failed 0.00
volume rebalance: test-replica-dist: success
# gluster volume heal test-replica-dist full
Launching heal operation to perform full self heal on volume test-replica-dist has been successful
Use heal info commands to check status
# gluster volume heal test-replica-dist info split-brain
Brick 10.10.60.169:/exports/test-dir/test-brick-1a
Number of entries in split-brain: 0
Brick 10.10.60.169:/exports/test-dir/test-brick-2a
Number of entries in split-brain: 0
Brick 10.10.60.169:/exports/test-dir/test-brick-1b
Number of entries in split-brain: 0
Brick 10.10.60.169:/exports/test-dir/test-brick-2b
Number of entries in split-brain: 0
# diff -r /lib64/ /mnt/test-replica-dist/lib64/
Only in /lib64/device-mapper: libdevmapper-event-lvm2thin.so
Only in /lib64/multipath: libcheckcciss_tur.so
Only in /lib64/multipath: libcheckemc_clariion.so
...
Only in /lib64/xtables: libxt_TRACE.so
Only in /lib64/xtables: libxt_udp.so
# view /var/log/glusterfs/test-replica-dist-rebalance.log
[2015-12-14 23:06:25.432546] E [dht-rebalance.c:2949:gf_defrag_fix_layout] 0-test-replica-dist-dht: //.trashcan gfid not present
[2015-12-14 23:06:25.433196] I [MSGID: 109081] [dht-common.c:3810:dht_setxattr] 0-test-replica-dist-dht: fixing the layout of /lib64
[2015-12-14 23:06:25.433217] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 0 (test-replica-dist-replicate-0): 1014 chunks
[2015-12-14 23:06:25.433228] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 1 (test-replica-dist-replicate-1): 1014 chunks
[2015-12-14 23:06:25.434564] I [dht-rebalance.c:2446:gf_defrag_process_dir] 0-test-replica-dist-dht: migrate data called on /lib64
[2015-12-14 23:06:25.562584] I [dht-rebalance.c:2656:gf_defrag_process_dir] 0-test-replica-dist-dht: Migration operation on dir /lib64 took 0.13 secs
[2015-12-14 23:06:25.568177] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables (0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc) [No data available]
[2015-12-14 23:06:25.568206] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables (0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc) [No data available]
[2015-12-14 23:06:25.568557] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.568581] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.569942] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.569964] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.570117] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-test-replica-dist-dht: Found anomalies in /lib64/xtables (gfid = 0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc). Holes=1 overlaps=0
[2015-12-14 23:06:25.570142] W [MSGID: 109005] [dht-selfheal.c:1805:dht_selfheal_directory] 0-test-replica-dist-dht: Directory selfheal failed : 1 subvolumes have unrecoverable errors. path = /lib64/xtables, gfid = 0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc
[2015-12-14 23:06:25.570179] I [MSGID: 109081] [dht-common.c:3810:dht_setxattr] 0-test-replica-dist-dht: fixing the layout of /lib64/xtables
[2015-12-14 23:06:25.570206] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 0 (test-replica-dist-replicate-0): 1014 chunks
[2015-12-14 23:06:25.570219] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 1 (test-replica-dist-replicate-1): 1014 chunks
[2015-12-14 23:06:25.570889] E [dht-rebalance.c:2992:gf_defrag_fix_layout] 0-test-replica-dist-dht: Setxattr failed for /lib64/xtables
[2015-12-14 23:06:25.571049] E [MSGID: 109016] [dht-rebalance.c:3006:gf_defrag_fix_layout] 0-test-replica-dist-dht: Fix layout failed for /lib64
[2015-12-14 23:06:25.571182] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 5
[2015-12-14 23:06:25.571255] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 6
[2015-12-14 23:06:25.571281] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 7
[2015-12-14 23:06:25.571302] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 8
[2015-12-14 23:06:25.571627] I [MSGID: 109028] [dht-rebalance.c:3485:gf_defrag_status_get] 0-test-replica-dist-dht: Rebalance is failed. Time taken is 0.00 secs
[2015-12-14 23:06:25.571647] I [MSGID: 109028] [dht-rebalance.c:3489:gf_defrag_status_get] 0-test-replica-dist-dht: Files migrated: 0, size: 0, lookups: 249, failures: 3, skipped: 0
# view /var/log/glusterfs/mnt-test-replica-dist-.log
[2015-12-14 23:05:03.797439] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.800523] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TPROXY.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.800626] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TPROXY.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.802003] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.802100] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.804886] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TRACE.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.804989] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TRACE.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.806342] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.806477] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.809396] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_u32.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.809430] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_u32.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.810841] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.810905] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.813705] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_udp.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.813824] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_udp.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.815201] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.815295] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.818854] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.818895] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.820314] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.820342] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.821992] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.822001] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.822207] W [fuse-resolve.c:65:fuse_resolve_entry_cbk] 0-fuse: 056bce17-a4c2-4e13-a352-40f783c4804a/ZIPScanner.so: failed to resolve (No data available)
[2015-12-14 23:05:03.822574] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.822600] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.824000] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.824076] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:01.386125] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64 (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:01.386151] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64 (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:01.411283] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/CFBScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:38.979973] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 8a2836d2-dd2c-46dc-9a1e-437c5444a704. (Possible split-brain)
[2015-12-14 23:07:38.983422] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 43c8e284-1cd0-48a8-b8e5-c075092eeaa7. (Possible split-brain)
[2015-12-14 23:07:39.031276] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 827357797
[2015-12-14 23:07:39.031587] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3874: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
[2015-12-14 23:07:39.032043] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/device-mapper (43c8e284-1cd0-48a8-b8e5-c075092eeaa7) [No data available]
[2015-12-14 23:07:39.032090] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/device-mapper (43c8e284-1cd0-48a8-b8e5-c075092eeaa7) [No data available]
[2015-12-14 23:07:39.033510] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.033523] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.034759] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.034790] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.035335] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3876: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
[2015-12-14 23:07:39.037553] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3881: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
[2015-12-14 23:07:39.037976] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3883: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
The message "W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 827357797" repeated 3 times between [2015-12-14 23:07:39.031276] and [2015-12-14 23:07:39.037686]
[2015-12-14 23:07:39.190729] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3553706931
[2015-12-14 23:07:39.190781] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3781782680
[2015-12-14 23:07:39.190990] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid eeab01c5-af5f-49f8-bd06-01471f405c84. (Possible split-brain)
[2015-12-14 23:07:39.214980] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid fe99280a-10f4-4e2f-8483-99d63461fa9e. (Possible split-brain)
[2015-12-14 23:07:39.227809] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3553706931
[2015-12-14 23:07:39.227837] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3781782680
[2015-12-14 23:07:39.228015] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 862461d1-38ef-4f3a-8216-c9d9dedde1af. (Possible split-brain)
[2015-12-14 23:07:39.264828] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3553706931
[2015-12-14 23:07:39.264862] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3781782680
[2015-12-14 23:07:39.266617] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc. (Possible split-brain
##########################################
# Successful "Distributed" recreate case.
##########################################
# mkdir /exports/test-dir/test-brick-1
# mkdir /exports/test-dir/test-brick-2
# gluster volume create test-dist transport tcp 10.10.60.169:/exports/test-dir/test-brick-1 10.10.60.169:/exports/test-dir/test-brick-2
volume create: test-dist: success: please start the volume to access data
# gluster volume start test-dist
volume start: test-dist: success
# gluster volume info test-dist
Volume Name: test-dist
Type: Distribute
Volume ID: 385a8546-1776-45be-8ae4-cd94ed37f2a5
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.10.60.169:/exports/test-dir/test-brick-1
Brick2: 10.10.60.169:/exports/test-dir/test-brick-2
Options Reconfigured:
snap-activate-on-create: enable
# mkdir /mnt/test-dist
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-dist /mnt/test-dist/
# cp -rf /lib64/ /mnt/test-dist/
# diff -r /lib64/ /mnt/test-dist/lib64/
# umount /mnt/test-dist/
# gluster volume stop test-dist
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test-dist: success
# gluster volume delete test-dist
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: test-dist: success
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-1
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1
xattr clean-up in progress: /exports/test-dir/test-brick-1
/exports/test-dir/test-brick-1 ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-2
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2
xattr clean-up in progress: /exports/test-dir/test-brick-2
/exports/test-dir/test-brick-2 ready to be used as a glusterfs brick
# gluster volume create test-dist transport tcp 10.10.60.169:/exports/test-dir/test-brick-1 10.10.60.169:/exports/test-dir/test-brick-2
volume create: test-dist: success: please start the volume to access data
# gluster volume start test-dist
volume start: test-dist: success
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-dist /mnt/test-dist/
# diff -r /lib64/ /mnt/test-dist/lib64/
# gluster volume rebalance test-dist start
volume rebalance: test-dist: success: Rebalance on test-dist has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: 63163e88-0b81-40cf-9050-4af12bf31acd
# gluster volume rebalance test-dist status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes 537 0 0 completed 1.00
volume rebalance: test-dist: success
##########################################
# Successful "Replicate" recreate case.
##########################################
# mkdir /exports/test-dir/test-brick-3a
# mkdir /exports/test-dir/test-brick-3b
# gluster volume create test-replica replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-3a 10.10.60.169:/exports/test-dir/test-brick-3b force volume create: test-replica: success: please start the volume to access data
# gluster volume start test-replica
volume start: test-replica: success
# gluster volume info test-replica
Volume Name: test-replica
Type: Replicate
Volume ID: 1e66af41-a29f-45ba-b25d-b4b16d2a66d9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.10.60.169:/exports/test-dir/test-brick-3a
Brick2: 10.10.60.169:/exports/test-dir/test-brick-3b
Options Reconfigured:
snap-activate-on-create: enable
# mkdir /mnt/test-replica
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica /mnt/test-replica
# cp -rf /lib64/ /mnt/test-replica/
# diff -r /lib64/ /mnt/test-replica/lib64/
# umount /mnt/test-replica
# gluster volume stop test-replica
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test-replica: success
# gluster volume delete test-replica
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: test-replica: success
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-3a
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-3a
xattr clean-up in progress: /exports/test-dir/test-brick-3a
/exports/test-dir/test-brick-3a ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-3b
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-3b
xattr clean-up in progress: /exports/test-dir/test-brick-3b
/exports/test-dir/test-brick-3b ready to be used as a glusterfs brick
# gluster volume create test-replica replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-3a 10.10.60.169:/exports/test-dir/test-brick-3b force
volume create: test-replica: success: please start the volume to access data
# gluster volume start test-replica
volume start: test-replica: success
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica /mnt/test-replica
# diff -r /lib64/ /mnt/test-replica/lib64/
GlusterFS volume, and then clean, and recreate it from the
component bricks, when the volume is "Distributed-Replicate".
The same steps used with a simple "Replicate" or "Distribute"
volume are successful with the correct data.
For the failing "Distributed-Replicate" case, the "rebalance"
always fails after the recreate. The "heal" command is
successful, showing no "info split-brain" files indicated, but
about half of the files are missing, and there are many
"(Possible split-brain)" warnings in the logfile.
The gluster "Distributed-Replicate" volume "recreate"
procedure works fine in GlusterFS versions 3.2.7, and 3.4.2,
but not in glusterfs 3.6.5, 3.6.7, or 3.7.6.
Perhaps the "recreate" procedure has changed, or I'm doing
something wrong that now matters in the newer GlusterFS
versions.
Details below. Any ideas how to make it work again?
Thanks.
~ Jeff Byers ~
# glusterd -V
glusterfs 3.7.6 built on Dec 14 2015 07:05:12
################################################
# Failing "Distributed-Replicate" recreate case.
################################################
# mountpoint /exports/test-dir/
/exports/test-dir/ is a mountpoint
# mount |grep test-dir
/dev/sdu on /exports/test-dir type xfs (rw,noatime,nodiratime,barrier,nouuid,inode64,logbufs=8,logbsize=256k)
# mkdir /exports/test-dir/test-brick-1a
# mkdir /exports/test-dir/test-brick-1b
# mkdir /exports/test-dir/test-brick-2a
# mkdir /exports/test-dir/test-brick-2b
# gluster volume create test-replica-dist replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-1a 10.10.60.169:/exports/test-dir/test-brick-2a 10.10.60.169:/exports/test-dir/test-brick-1b 10.10.60.169:/exports/test-dir/test-brick-2b force
volume create: test-replica-dist: success: please start the volume to access data
# gluster volume start test-replica-dist
volume start: test-replica-dist: success
# gluster volume info test-replica-dist
Volume Name: test-replica-dist
Type: Distributed-Replicate
Volume ID: c8de4e65-2304-4801-a244-6511f39fc0c9
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.10.60.169:/exports/test-dir/test-brick-1a
Brick2: 10.10.60.169:/exports/test-dir/test-brick-2a
Brick3: 10.10.60.169:/exports/test-dir/test-brick-1b
Brick4: 10.10.60.169:/exports/test-dir/test-brick-2b
Options Reconfigured:
snap-activate-on-create: enable
# mkdir /mnt/test-replica-dist
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica-dist /mnt/test-replica-dist/
# cp -rf /lib64/ /mnt/test-replica-dist/
# diff -r /lib64/ /mnt/test-replica-dist/lib64/
# umount /mnt/test-replica-dist
# gluster volume stop test-replica-dist
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test-replica-dist: success
# gluster volume delete test-replica-dist
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: test-replica-dist: success
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-1a
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1a
xattr clean-up in progress: /exports/test-dir/test-brick-1a
/exports/test-dir/test-brick-1a ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-1b
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1b
xattr clean-up in progress: /exports/test-dir/test-brick-1b
/exports/test-dir/test-brick-1b ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-2a
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2a
xattr clean-up in progress: /exports/test-dir/test-brick-2a
/exports/test-dir/test-brick-2a ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-2b
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2b
xattr clean-up in progress: /exports/test-dir/test-brick-2b
/exports/test-dir/test-brick-2b ready to be used as a glusterfs brick
# gluster volume create test-replica-dist replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-1a 10.10.60.169:/exports/test-dir/test-brick-2a 10.10.60.169:/exports/test-dir/test-brick-1b 10.10.60.169:/exports/test-dir/test-brick-2b force
volume create: test-replica-dist: success: please start the volume to access data
# gluster volume start test-replica-dist
volume start: test-replica-dist: success
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica-dist /mnt/test-replica-dist/
# diff -r /lib64/ /mnt/test-replica-dist/lib64/
Only in /lib64/device-mapper: libdevmapper-event-lvm2thin.so
Only in /lib64/multipath: libcheckcciss_tur.so
Only in /lib64/multipath: libcheckemc_clariion.so
Only in /lib64/multipath: libcheckhp_sw.so
Only in /lib64/multipath: libprioconst.so
Only in /lib64/multipath: libpriordac.so
Only in /lib64/multipath: libprioweighted.so
Only in /lib64/rtkaio: librtkaio-2.12.so
Only in /lib64/rtkaio: librt.so.1
Only in /lib64/xtables: libip6t_ah.so
Only in /lib64/xtables: libip6t_dst.so
Only in /lib64/xtables: libip6t_eui64.so
Only in /lib64/xtables: libip6t_frag.so
Only in /lib64/xtables: libip6t_HL.so
Only in /lib64/xtables: libip6t_icmp6.so
Only in /lib64/xtables: libip6t_LOG.so
Only in /lib64/xtables: libip6t_mh.so
Only in /lib64/xtables: libip6t_REJECT.so
Only in /lib64/xtables: libip6t_set.so
Only in /lib64/xtables: libipt_ah.so
Only in /lib64/xtables: libipt_ecn.so
Only in /lib64/xtables: libipt_ECN.so
Only in /lib64/xtables: libipt_icmp.so
Only in /lib64/xtables: libipt_MIRROR.so
Only in /lib64/xtables: libipt_realm.so
Only in /lib64/xtables: libipt_REDIRECT.so
Only in /lib64/xtables: libipt_REJECT.so
Only in /lib64/xtables: libipt_SAME.so
Only in /lib64/xtables: libipt_SET.so
Only in /lib64/xtables: libipt_SNAT.so
Only in /lib64/xtables: libipt_ttl.so
Only in /lib64/xtables: libipt_ULOG.so
Only in /lib64/xtables: libipt_unclean.so
Only in /lib64/xtables: libxt_CHECKSUM.so
Only in /lib64/xtables: libxt_cluster.so
Only in /lib64/xtables: libxt_connbytes.so
Only in /lib64/xtables: libxt_connlimit.so
Only in /lib64/xtables: libxt_CONNMARK.so
Only in /lib64/xtables: libxt_CONNSECMARK.so
Only in /lib64/xtables: libxt_conntrack.so
Only in /lib64/xtables: libxt_dccp.so
Only in /lib64/xtables: libxt_dscp.so
Only in /lib64/xtables: libxt_iprange.so
Only in /lib64/xtables: libxt_length.so
Only in /lib64/xtables: libxt_limit.so
Only in /lib64/xtables: libxt_mac.so
Only in /lib64/xtables: libxt_multiport.so
Only in /lib64/xtables: libxt_osf.so
Only in /lib64/xtables: libxt_physdev.so
Only in /lib64/xtables: libxt_pkttype.so
Only in /lib64/xtables: libxt_policy.so
Only in /lib64/xtables: libxt_quota.so
Only in /lib64/xtables: libxt_RATEEST.so
Only in /lib64/xtables: libxt_sctp.so
Only in /lib64/xtables: libxt_SECMARK.so
Only in /lib64/xtables: libxt_socket.so
Only in /lib64/xtables: libxt_statistic.so
Only in /lib64/xtables: libxt_string.so
Only in /lib64/xtables: libxt_tcpmss.so
Only in /lib64/xtables: libxt_TCPOPTSTRIP.so
Only in /lib64/xtables: libxt_time.so
Only in /lib64/xtables: libxt_TOS.so
Only in /lib64/xtables: libxt_TPROXY.so
Only in /lib64/xtables: libxt_TRACE.so
Only in /lib64/xtables: libxt_udp.so
# gluster volume rebalance test-replica-dist start
volume rebalance: test-replica-dist: success: Rebalance on test-replica-dist has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: ccf76757-c3df-4ae2-af2d-b82f8283d821
# gluster volume rebalance test-replica-dist status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes 249 3 0 failed 0.00
volume rebalance: test-replica-dist: success
# gluster volume heal test-replica-dist full
Launching heal operation to perform full self heal on volume test-replica-dist has been successful
Use heal info commands to check status
# gluster volume heal test-replica-dist info split-brain
Brick 10.10.60.169:/exports/test-dir/test-brick-1a
Number of entries in split-brain: 0
Brick 10.10.60.169:/exports/test-dir/test-brick-2a
Number of entries in split-brain: 0
Brick 10.10.60.169:/exports/test-dir/test-brick-1b
Number of entries in split-brain: 0
Brick 10.10.60.169:/exports/test-dir/test-brick-2b
Number of entries in split-brain: 0
# diff -r /lib64/ /mnt/test-replica-dist/lib64/
Only in /lib64/device-mapper: libdevmapper-event-lvm2thin.so
Only in /lib64/multipath: libcheckcciss_tur.so
Only in /lib64/multipath: libcheckemc_clariion.so
...
Only in /lib64/xtables: libxt_TRACE.so
Only in /lib64/xtables: libxt_udp.so
# view /var/log/glusterfs/test-replica-dist-rebalance.log
[2015-12-14 23:06:25.432546] E [dht-rebalance.c:2949:gf_defrag_fix_layout] 0-test-replica-dist-dht: //.trashcan gfid not present
[2015-12-14 23:06:25.433196] I [MSGID: 109081] [dht-common.c:3810:dht_setxattr] 0-test-replica-dist-dht: fixing the layout of /lib64
[2015-12-14 23:06:25.433217] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 0 (test-replica-dist-replicate-0): 1014 chunks
[2015-12-14 23:06:25.433228] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 1 (test-replica-dist-replicate-1): 1014 chunks
[2015-12-14 23:06:25.434564] I [dht-rebalance.c:2446:gf_defrag_process_dir] 0-test-replica-dist-dht: migrate data called on /lib64
[2015-12-14 23:06:25.562584] I [dht-rebalance.c:2656:gf_defrag_process_dir] 0-test-replica-dist-dht: Migration operation on dir /lib64 took 0.13 secs
[2015-12-14 23:06:25.568177] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables (0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc) [No data available]
[2015-12-14 23:06:25.568206] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables (0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc) [No data available]
[2015-12-14 23:06:25.568557] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.568581] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.569942] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.569964] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:06:25.570117] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-test-replica-dist-dht: Found anomalies in /lib64/xtables (gfid = 0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc). Holes=1 overlaps=0
[2015-12-14 23:06:25.570142] W [MSGID: 109005] [dht-selfheal.c:1805:dht_selfheal_directory] 0-test-replica-dist-dht: Directory selfheal failed : 1 subvolumes have unrecoverable errors. path = /lib64/xtables, gfid = 0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc
[2015-12-14 23:06:25.570179] I [MSGID: 109081] [dht-common.c:3810:dht_setxattr] 0-test-replica-dist-dht: fixing the layout of /lib64/xtables
[2015-12-14 23:06:25.570206] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 0 (test-replica-dist-replicate-0): 1014 chunks
[2015-12-14 23:06:25.570219] I [MSGID: 109045] [dht-selfheal.c:1509:dht_fix_layout_of_directory] 0-test-replica-dist-dht: subvolume 1 (test-replica-dist-replicate-1): 1014 chunks
[2015-12-14 23:06:25.570889] E [dht-rebalance.c:2992:gf_defrag_fix_layout] 0-test-replica-dist-dht: Setxattr failed for /lib64/xtables
[2015-12-14 23:06:25.571049] E [MSGID: 109016] [dht-rebalance.c:3006:gf_defrag_fix_layout] 0-test-replica-dist-dht: Fix layout failed for /lib64
[2015-12-14 23:06:25.571182] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 5
[2015-12-14 23:06:25.571255] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 6
[2015-12-14 23:06:25.571281] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 7
[2015-12-14 23:06:25.571302] I [dht-rebalance.c:2085:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 8
[2015-12-14 23:06:25.571627] I [MSGID: 109028] [dht-rebalance.c:3485:gf_defrag_status_get] 0-test-replica-dist-dht: Rebalance is failed. Time taken is 0.00 secs
[2015-12-14 23:06:25.571647] I [MSGID: 109028] [dht-rebalance.c:3489:gf_defrag_status_get] 0-test-replica-dist-dht: Files migrated: 0, size: 0, lookups: 249, failures: 3, skipped: 0
# view /var/log/glusterfs/mnt-test-replica-dist-.log
[2015-12-14 23:05:03.797439] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.800523] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TPROXY.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.800626] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TPROXY.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.802003] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.802100] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.804886] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TRACE.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.804989] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_TRACE.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.806342] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.806477] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.809396] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_u32.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.809430] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_u32.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.810841] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.810905] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.813705] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_udp.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.813824] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/xtables-1.4.7/libxt_udp.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.815201] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.815295] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.818854] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.818895] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.820314] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.820342] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.821992] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.822001] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.822207] W [fuse-resolve.c:65:fuse_resolve_entry_cbk] 0-fuse: 056bce17-a4c2-4e13-a352-40f783c4804a/ZIPScanner.so: failed to resolve (No data available)
[2015-12-14 23:05:03.822574] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.822600] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: /lib64/ZIPScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.824000] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:03.824076] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:01.386125] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64 (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:01.386151] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64 (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:05:01.411283] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/CFBScanner.so (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:38.979973] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 8a2836d2-dd2c-46dc-9a1e-437c5444a704. (Possible split-brain)
[2015-12-14 23:07:38.983422] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 43c8e284-1cd0-48a8-b8e5-c075092eeaa7. (Possible split-brain)
[2015-12-14 23:07:39.031276] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 827357797
[2015-12-14 23:07:39.031587] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3874: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
[2015-12-14 23:07:39.032043] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: /lib64/device-mapper (43c8e284-1cd0-48a8-b8e5-c075092eeaa7) [No data available]
[2015-12-14 23:07:39.032090] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: /lib64/device-mapper (43c8e284-1cd0-48a8-b8e5-c075092eeaa7) [No data available]
[2015-12-14 23:07:39.033510] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.033523] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.034759] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.034790] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-test-replica-dist-client-3: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2015-12-14 23:07:39.035335] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3876: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
[2015-12-14 23:07:39.037553] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3881: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
[2015-12-14 23:07:39.037976] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 3883: LOOKUP() /lib64/device-mapper/libdevmapper-event-lvm2thin.so => -1 (Stale file handle)
The message "W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 827357797" repeated 3 times between [2015-12-14 23:07:39.031276] and [2015-12-14 23:07:39.037686]
[2015-12-14 23:07:39.190729] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3553706931
[2015-12-14 23:07:39.190781] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3781782680
[2015-12-14 23:07:39.190990] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid eeab01c5-af5f-49f8-bd06-01471f405c84. (Possible split-brain)
[2015-12-14 23:07:39.214980] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid fe99280a-10f4-4e2f-8483-99d63461fa9e. (Possible split-brain)
[2015-12-14 23:07:39.227809] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3553706931
[2015-12-14 23:07:39.227837] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3781782680
[2015-12-14 23:07:39.228015] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 862461d1-38ef-4f3a-8216-c9d9dedde1af. (Possible split-brain)
[2015-12-14 23:07:39.264828] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3553706931
[2015-12-14 23:07:39.264862] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-test-replica-dist-dht: no subvolume for hash (value) = 3781782680
[2015-12-14 23:07:39.266617] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-test-replica-dist-replicate-1: Unreadable subvolume -1 found with event generation 2 for gfid 0e2a4e5c-7c8b-4f8b-979a-9dbd73de6ecc. (Possible split-brain
##########################################
# Successful "Distributed" recreate case.
##########################################
# mkdir /exports/test-dir/test-brick-1
# mkdir /exports/test-dir/test-brick-2
# gluster volume create test-dist transport tcp 10.10.60.169:/exports/test-dir/test-brick-1 10.10.60.169:/exports/test-dir/test-brick-2
volume create: test-dist: success: please start the volume to access data
# gluster volume start test-dist
volume start: test-dist: success
# gluster volume info test-dist
Volume Name: test-dist
Type: Distribute
Volume ID: 385a8546-1776-45be-8ae4-cd94ed37f2a5
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.10.60.169:/exports/test-dir/test-brick-1
Brick2: 10.10.60.169:/exports/test-dir/test-brick-2
Options Reconfigured:
snap-activate-on-create: enable
# mkdir /mnt/test-dist
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-dist /mnt/test-dist/
# cp -rf /lib64/ /mnt/test-dist/
# diff -r /lib64/ /mnt/test-dist/lib64/
# umount /mnt/test-dist/
# gluster volume stop test-dist
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test-dist: success
# gluster volume delete test-dist
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: test-dist: success
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-1
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1
xattr clean-up in progress: /exports/test-dir/test-brick-1
/exports/test-dir/test-brick-1 ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-2
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2
xattr clean-up in progress: /exports/test-dir/test-brick-2
/exports/test-dir/test-brick-2 ready to be used as a glusterfs brick
# gluster volume create test-dist transport tcp 10.10.60.169:/exports/test-dir/test-brick-1 10.10.60.169:/exports/test-dir/test-brick-2
volume create: test-dist: success: please start the volume to access data
# gluster volume start test-dist
volume start: test-dist: success
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-dist /mnt/test-dist/
# diff -r /lib64/ /mnt/test-dist/lib64/
# gluster volume rebalance test-dist start
volume rebalance: test-dist: success: Rebalance on test-dist has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: 63163e88-0b81-40cf-9050-4af12bf31acd
# gluster volume rebalance test-dist status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes 537 0 0 completed 1.00
volume rebalance: test-dist: success
##########################################
# Successful "Replicate" recreate case.
##########################################
# mkdir /exports/test-dir/test-brick-3a
# mkdir /exports/test-dir/test-brick-3b
# gluster volume create test-replica replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-3a 10.10.60.169:/exports/test-dir/test-brick-3b force volume create: test-replica: success: please start the volume to access data
# gluster volume start test-replica
volume start: test-replica: success
# gluster volume info test-replica
Volume Name: test-replica
Type: Replicate
Volume ID: 1e66af41-a29f-45ba-b25d-b4b16d2a66d9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.10.60.169:/exports/test-dir/test-brick-3a
Brick2: 10.10.60.169:/exports/test-dir/test-brick-3b
Options Reconfigured:
snap-activate-on-create: enable
# mkdir /mnt/test-replica
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica /mnt/test-replica
# cp -rf /lib64/ /mnt/test-replica/
# diff -r /lib64/ /mnt/test-replica/lib64/
# umount /mnt/test-replica
# gluster volume stop test-replica
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test-replica: success
# gluster volume delete test-replica
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: test-replica: success
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-3a
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-3a
xattr clean-up in progress: /exports/test-dir/test-brick-3a
/exports/test-dir/test-brick-3a ready to be used as a glusterfs brick
# gluster_clear_xattrs.sh /exports/test-dir/test-brick-3b
removing all .glusterfs directories in progress: /exports/test-dir/test-brick-3b
xattr clean-up in progress: /exports/test-dir/test-brick-3b
/exports/test-dir/test-brick-3b ready to be used as a glusterfs brick
# gluster volume create test-replica replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-3a 10.10.60.169:/exports/test-dir/test-brick-3b force
volume create: test-replica: success: please start the volume to access data
# gluster volume start test-replica
volume start: test-replica: success
# mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica /mnt/test-replica
# diff -r /lib64/ /mnt/test-replica/lib64/
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
