Re: Best practices for copying lots of files machine-to-machine

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



On 17 May 2017 at 22:27, <m.roth@xxxxxxxxx> wrote:

> Vanhorn, Mike wrote:
> > On 5/17/17, 12:03 PM, "CentOS on behalf of ken" <
> centos-bounces@xxxxxxxxxx
> > on behalf of gebser@xxxxxxxxxxxx> wrote:
> >
> >>An entire filesystem (~180g) needs to be copied from one local linux
> >>machine to another.  Since both systems are on the same local subnet,
> >>there's no need for encryption.
> >>
> >>I've done this sort of thing before a few times in the past in different
> >>ways, but wanted to get input from others on what's worked best for
> >> them.
> >
> > If shutting the machines down is feasible, I’d put the source hard drive
> > into the destination machine and use rsync to copy it from one drive to
> > the other (rather than using rsync to copy from one machine to the other
> > over the network).
>
> Why? I just rsync'd 159G in less than one workday from one server to
> another. Admittedly, we allegedly have a 1G network, but....
>
>        mark
>
> Hi,
you can parallelize rsync with xargs's -P (max-procs) option (man xargs).

rsync -a -f"+ */" -f"- *" source/ server:/destination/  #sync directory
first
cd source/; find . -type f | *xargs* -n1 -*P0* -I% rsync -az %
server:/destination/% # 0 to let xargs deal with the num of procs

Julius
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos




[Index of Archives]     [CentOS]     [CentOS Announce]     [CentOS Development]     [CentOS ARM Devel]     [CentOS Docs]     [CentOS Virtualization]     [Carrier Grade Linux]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [Xorg]     [Linux USB]
  Powered by Linux