On Sat, Jul 19, 2014 at 10:02:33AM -0400, Benjamin Turner wrote: > On Fri, Jul 18, 2014 at 10:43 PM, Pranith Kumar Karampuri < > pkarampu@xxxxxxxxxx> wrote: > > > > > On 07/18/2014 07:57 PM, Anders Blomdell wrote: > > > >> During testing of a 3*4 gluster (from master as of yesterday), I > >> encountered > >> two major weirdnesses: > >> > >> 1. A 'rm -rf <some_dir>' needed several invocations to finish, each > >> time > >> reporting a number of lines like these: > >> rm: cannot remove ‘a/b/c/d/e/f’: Directory not empty > >> > > > This is reproducible for me when running dbench on nfs mounts. I think I > may have seen it on glusterfs mounts as well but it seems more reproducible > on nfs. I should have caught it sooner but it doesn't error out client > side when cleaning up, and the next test I run the deletes are successful. > When this happens in the nfs.log I see: > > This spams the log, from what I can tell it happens when dbench is creating > the files: > [2014-07-19 13:37:03.271651] I [MSGID: 109036] > [dht-common.c:5694:dht_log_new_layout_for_dir_selfheal] 0-testvol-dht: > Setting layout of /clients/client3/~dmtmp/SEED with [Subvol_name: > testvol-replicate-0, Err: -1 , Start: 2147483647 , Stop: 4294967295 ], > [Subvol_name: testvol-replicate-1, Err: -1 , Start: 0 , Stop: 2147483646 ], My guess is that DHT/AFR fail to create the whole directory structure on all bricks (remember that directories should get created on all bricks, even for a dht only volume). If creating a directory fails on a particular brick, self-heal should pick it up... But well, maybe self-heal is not run when deleting directories, causing some directories on some bricks to be non-empty, but empty on others. It may be that this conflict is not handled correctly. You could maybe test with different volumes, and narrow down where the issue occurs: - a volume of one brick - a replicate volume with two bricks - a distribute volume with two bricks Potentially increase the number of bricks when a 2-brick afr-only, or dht-only volume does not trigger the issue reliably or quickly. > Occasionally I see: > [2014-07-19 13:50:46.091429] W [socket.c:529:__socket_rwv] 0-NLM-client: > readv on 192.168.11.102:45823 failed (No data available) > [2014-07-19 13:50:46.091570] E [rpc-transport.c:485:rpc_transport_unref] > (-->/usr/lib64/glusterfs/3.5qa2/xlator/nfs/server.so(nlm_rpcclnt_notify+0x5a) > [0x7f53775128ea] > (-->/usr/lib64/glusterfs/3.5qa2/xlator/nfs/server.so(nlm_unset_rpc_clnt+0x75) > [0x7f537750e3e5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) > [0x7f5388914693]))) 0-rpc_transport: invalid argument: this This looks like a bug in the NFS-server, I suggest to file is independently from the directory tree create/delete issue. > I'm opening a BZ now, I'll leave systems up and put the repro steps + > hostnames in the BZ in case anyone wants to poke around. Thanks! The NFS problem does not need any checking on the running system. Niels > > -b > > > > > > >> 2. After having successfully deleted all files from the volume, > >> i have a single directory that is duplicated in gluster-fuse, > >> like this: > >> # ls -l /mnt/gluster > >> total 24 > >> drwxr-xr-x 2 root root 12288 18 jul 16.17 work2/ > >> drwxr-xr-x 2 root root 12288 18 jul 16.17 work2/ > >> > >> any idea on how to debug this issue? > >> > > What are the steps to recreate? We need to first find what lead to this. > > Then probably which xlator leads to this. > > > > I have not seen this but I am running on a 6x2 volume. I wonder if this > may only happen with replica > 2? > > > > > > Pranith > > > >> > >> /Anders > >> > >> > > > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@xxxxxxxxxxx > > http://supercolony.gluster.org/mailman/listinfo/gluster-devel > > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxxx > http://supercolony.gluster.org/mailman/listinfo/gluster-devel _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-devel
