All - Recently one of bricks lost a hard drive, during the rebuild (3ware 9690 controller) we lost another drive and then had a few ECC errors on a third. This was on a ~30tb 24 drive RAID6 array. I was able to force the controller to rebuild with a ignore_ECC flag which has completed successfully and the XFS partition appears to be fine. In the 3ware device logs I see a dozen alerts about bad sectors/ecc errors. The partition is at 97% full so its a pretty good chance we have some data corruption. But not to worry, Gluster to the rescue right?? We currently have two copies of our data and gluster handles the replication between them - let's call them "A" and "B" clusters. Our "A" cluster is the cluster having issues. We've been planning on adding a third "C" cluster for extra reliability and mostly for the added performance. So since I have a working good copy of the brick that is having issues on our "B" cluster I started a gluster sync of our "B" cluster to the new "C" cluster. And OMG its so slow. I've been running a "ls -alR" for the last week and its only done 3.8% (replicated ~9.9 million files) of our total space with an estimate finish date of another 223 days - thats the end of August! So my question is how can I get this done quicker? Can I rsync one brick to another brick directly - I know that will not copy the extended attributes correctly and I believe will mess up gluster right? Anybody have some great ideas? I'm running gluster 2.0.9 with 64bit Centos 5.7/6.2. Each A/B/C cluster is 4 x 30tb xfs raid6 bricks for a total of ~120tb (84tb in use). liam
