That's good to hear, Dan. Yeah, since I published that I've heard a variety of similar solutions (like with GNU parallel). It's really good to get the information out there, because you never know you need to run rsync with multiple threads until you need it! I think the more solutions we have, the better. Mine makes sense to me, because I went through the whole ordeal and built it step by step, but I can see how other methods would make more sense to other people. That's also why I attempted to explain all the rsync and xargs options on my blog post! Regards, Alan On 07/10/2014 04:27 AM, Dan Mons wrote: > We do something similar for our nightly backups (100TB between two > Gluster setups). > > Each of our 6 Gluster nodes gets a set of top level folders > (representing each department in the org), and within each we thread > based on folders in the top level of each major section. That nets us > around 200+ rsync threads, which makes the nightly sync happen a lot > faster. > > I played around with parallel rsync, but could never make it work the > way I wanted. Just doing a simple "ls -d * | while read DIR ; do > rsync /$DIR/ remote:/$DIR/ & done" works out far better. > > -Dan > > ---------------- > Dan Mons > Unbreaker of broken things > Cutting Edge > http://cuttingedge.com.au > > > On 9 July 2014 21:42, Alan Orth <alan.orth@xxxxxxxxx> wrote: >> Hi, >> >> I recently had a RAID failure on one of my Gluster replicas; luckily my >> replica was ok, and I could re-sync all the data to the bad node's >> bricks. I used rsync to pre-seed the brick data, rather than having >> Gluster's self-heal daemon try to figure it out. >> >> It turns out I had way more files than I realized, which exposed some >> problems with "traditional" rsync invocation. I found some clever ways >> to optimize the transfer and speed up the process, and wrote up my >> experiences on my blog: >> >> http://mjanja.co.ke/2014/07/parallelizing-rsync/ >> >> Hope this helps someone! >> >> -- >> Alan Orth >> alan.orth@xxxxxxxxx >> http://alaninkenya.org >> http://mjanja.co.ke >> "I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone." -Bjarne Stroustrup, inventor of C++ >> GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0 >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Alan Orth alan.orth@xxxxxxxxx http://alaninkenya.org http://mjanja.co.ke "I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone." -Bjarne Stroustrup, inventor of C++ GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
