Hi Craig, The md5sum of the files was indeed the same, which is why I thought it was very peculiar. Basically, all my servers are clients & bricks, in an AFR configuration. The goal is to replicate the data as much as possible so it's available locally on all machines. However - I went through the configuration files on all the machines, and it seems I've neglected to remove some of the bricks from the volumes, so there was a situation where some of the clients kept trying to save on bricks they were no longer supposed to be a part of. I have now ensured proper separation and configuration, and I am no longer getting the split brain messages in the log, at least for now. - I'm using the Gluster version that comes with Ubuntu 10.04 Server, which is version 3.0.2. It's exactly the same OS version & kernel, and Gluster version, on all of my machines. - This is my kernel: Linux some.machine.com 2.6.32-24-server #38-Ubuntu SMP Mon Jul 5 10:29:32 UTC 2010 x86_64 GNU/Linux - The on-disk FS is ext4. - I use glusterfs via a mount line in /etc/fstab What I have yet to figure out, is how to get the files to be read at "local" speed. I mean, seeing as the files are all physically in /data/export/ on ALL my servers, is there something I can do to make the reads really fast? The writes are very rare, so I really don't care much about writes. Thanks! On Aug 4, 2010, at 6:04 AM, Craig Carl wrote: > CountZ - > Some questions to help us troubleshoot - > > What version of Gluster? > What is your server /version/kernel? > What is the on-disk filesystem? > Can you md5sum the file on all the storage servers? Is it the same everywhere? > > How are you exporting Gluster (Gluster FS, NFS, CIFS, FTP, etc?) > What version of Gluster on the clients, if using it? > What are your clients disti/version/kernel? > If you md5sum the file from a couple of clients are the results the same across the clients and the servers? > > Thanks for your help. > > Craig > 408-829-9953 > Gluster Inc. > > > Sent from a mobile device, please excuse my tpyos. > > On Aug 3, 2010, at 20:32, Count Zero <countz at gmail.com> wrote: > >> It's local and is a standard AFR setup. I believe all the files are actually the same, but I'll verify this again. It just does this for a LOT of files, and they are all the same files (nothing has changed really). >> >> About WAN: I have mostly given up on WAN replication at the moment, so I use glusterfs for local groups of machines that are on the same switch, and I use a separate solution to sync between WAN glusters. >> >> So how do I delete without erasing the file from the entire gluster? >> >> I'm assuming I need to: >> >> 1) Unmount all the clients >> 2) Erase and recreate /data/export on all nodes other than the chosen "master" >> 3) Remount the clients, and access the files >> >> Is that right? >> >> >> On Aug 4, 2010, at 4:14 AM, Tejas N. Bhise wrote: >> >>> Is this over the WAN replicated setup ? Or a local setup ? >>> >>> ----- Original Message ----- >>> From: "Count Zero" <countz at gmail.com> >>> To: "Gluster General Discussion List" <gluster-users at gluster.org> >>> Sent: Wednesday, August 4, 2010 8:38:02 AM >>> Subject: Split Brain? >>> >>> I am seeing a lot of those in my cluster client's log file: >>> >>> [2010-08-04 04:06:30] E [afr-self-heal-data.c:705:afr_sh_data_fix] replicate: Unable to self-heal contents of '/lib/wms-server.jar' (possible split-brain). Please delete the file from all but the preferred subvolume. >>> >>> How do I recover from this without losing my files? >>> >>> Thanks, >>> CountZ >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
