Having an odd problem on a new test environment we are setting up for a
partner. And not sure where to look next to figure out the problem or
really understand what the dht_layout_dir_mismatch INFO message is
telling me.
I was turning up a 4 node distributed volume, each brick is its own
19TB ext4 partition on a hardware raid5. Each node has the volume
mounted back to itself at /glusterfs via the fuse mount.
At some point files I copy to the /glusterfs on Node A, are not
viewable at /glusterfs on the other 3 Nodes. Once that starts happening
I notice the below log messages generated anytime i try manipulating
those paths. When I try to remove a directory doing this from any node
with 'rm -rf /glusterfs/users/TARGETDIR' it fails to remove any of the
files stored on Node A. Far as I can tell none of the glusterd services
are down, or unresponsive.
The only thing I did different on this setup is that I recreated and
destroyed the gluster volume a couple times before hand. Stopping and
deleting the gluster volume with the cli, umount'ing the brick's
partition and rerunning mkfs.ext4 between each time so the filesystem
would be clean. If you delete and recreate the volume it will continue
to do the above.
Ubuntu 12.04 LTS
Linux kg14-compute-64 3.2.0-52-generic #78-Ubuntu SMP Fri Jul 26
16:21:44 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
root at kg14-compute-64:/home/lacadmin# dpkg -l | grep gluster
ii glusterfs-client 3.4.0final-ubuntu1~precise1
clustered file-system (client package)
ii glusterfs-common 3.4.0final-ubuntu1~precise1
GlusterFS common libraries and translator modules
ii glusterfs-server 3.4.0final-ubuntu1~precise1
clustered file-system (server package)
/etc/fstab
------
/dev/sdb1 /exports/gluster ext4
defaults,noatime,acl,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0
0 0
localhost:/USER-HOME /glusterfs glusterfs defaults,noauto,nobootwait 0 0
glusterfs.log
------
[2013-09-18 21:48:54.686845] I [dht-common.c:623:dht_revalidate_cbk]
0-USER-HOME-dht: mismatching layouts for /users/rpowell1/benchmark
[2013-09-18 21:48:54.687492] I [dht-layout.c:630:dht_layout_normalize]
0-USER-HOME-dht: found anomalies in /users/rpowell1/benchmark. holes=1
overlaps=1
[2013-09-18 22:04:32.671426] W [socket.c:514:__socket_rwv] 0-glusterfs:
readv failed (No data available)
[2013-09-18 22:04:32.671484] W
[socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from
socket failed. Error (No data available), peer (127.0.0.1:24007)
[2013-09-18 22:04:42.977516] I [glusterfsd-mgmt.c:1583:mgmt_getspec_cbk]
0-glusterfs: No change in volfile, continuing
[2013-09-18 22:14:25.221279] I
[dht-layout.c:745:dht_layout_dir_mismatch] 0-USER-HOME-dht: subvol:
USER-HOME-client-2; inode layout - 2147483646 - 3221225468; disk layout
- 0 - 1073741822
[2013-09-18 22:14:25.221338] I [dht-common.c:623:dht_revalidate_cbk]
0-USER-HOME-dht: mismatching layouts for /
[2013-09-18 22:14:25.221975] I [dht-layout.c:630:dht_layout_normalize]
0-USER-HOME-dht: found anomalies in /. holes=1 overlaps=1
[2013-09-19 00:23:11.341443] I
[dht-layout.c:745:dht_layout_dir_mismatch] 0-USER-HOME-dht: subvol:
USER-HOME-client-2; inode layout - 2147483646 - 3221225468; disk layout
- 0 - 1073741822
--
Raymond Powell
Systems Administrator
Laboratory for Advanced Computing
University of Chicago, KCBD 10160
900 East 57th Street
Chicago, IL 60637
