are you using ext4 with redhat/centos? There is a previous thread that shows some kind of bug with ext4 that causes similar sounding problems. If you are using ext4, try using xfs. On Tue, Aug 14, 2012 at 11:12 PM, Brian Candler <B.Candler at pobox.com> wrote: > On Tue, Aug 14, 2012 at 08:19:27PM -0700, stephen pierce wrote: >> I let both clients run for a while, then I stop one client. I then >> reset the brick/server that is not active (the other one is servicing >> the HTTP traffic) now. > > Do you mean that client1 sends HTTP traffic to brick/server1, and client2 > sends HTTP traffic to brick/server2? > >> While investigating, I discover that there are a lot of phantom >> files that are listed with just a filename, and lots of question marks >> (????) when doing an ls l. rm rf * on the Gluster volume seems to >> complete, but leaves behind all the broken files. > > It would be helpful if you could show the actual ls -l output, but my guess > is you are seeing something like this (demo on a local filesystem, not > gluster): > > $ mkdir testdir > $ touch testdir/testfile > $ chmod -x testdir > $ ls -l testdir > ls: cannot access testdir/testfile: Permission denied > total 0 > -????????? ? ? ? ? ? testfile > > If so, these aren't really "phantom files", but the permissions of the > enclosing directory are set wrongly (which might be some intermediate state > in gluster replication, I don't know) > > So an "ls -ld" of the parent directory would also be a good thing. Also, are > these filenames those you'd expect your application to create? > > What might be helpful is to trace your backend-application and what's making > it return a 500 server error, which may or may not be related to these > permissions. If you can see what file operations the backend is trying to > do and what filesystem error is being returned (e.g. with strace), this may > make it clearer what's going on. Then you can perhaps crank up gluster logs > at the appropriate place too. > > Any log messages talking about "split brain" would be especially interesting. > > Regards, > > Brian. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users