Re: rsync or dd to clone a hard drive?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Kwan Lowe wrote:
> On Wed, Oct 6, 2010 at 12:23 PM, Patrick Bartek<bartek047@xxxxxxxxx>  wrote:
>> I would use dd to clone (or back up) an entire hard drive.  Easier.  You can even pipe it through gzip to get a compressed image file.
>
> I do exactly that... dd piped through gzip then push through an SSH
> session to a remote where it's extracted and written on the fly to
> another LV.  I use it to move kvm LVs from machine to machine.  Works
> great, but there's not much error checking.  If it's critical I do an
> md5sum on the LVs at the end then compare.

Unless you have a really fast CPU or a really slow network the compression it 
taking longer than the network transfer. And in many cases you don't need the 
encryption of ssh, it just uses more CPU.

What to do?

One thing which can help is to use the built-in compression of ssh, depending on 
your setup that might be faster than gzip. But having the compress and encrypt 
in two cores helps with CPU usage. Another, if you are going through a trusted 
physical network (like a piece of cable or internal network) is to use netcat 
(aka "nc") for a full speed unencrypted transfer. Use common sense, sometimes 
security is needed. Finally, the lzop program uses way less CPU per byte than 
gzip, its compression sucks, but if the compression is running slower than the 
network lzop is usually a win, and if you need encryption the compress and 
encrypt are in separate threads (and hopefully cores).

Beware pipes, which are slow in general. Piping your dd into ssh can slow the 
transfer by a factor of five.[1]

Finally, if you have a good connection, enable jumbo packets in the routing at 
both ends, that will buy you some improved network speed and lower overhead.

I move a few TB between systems each week, once a week, so I've played with this 
a bit.

[1] simple pipe test:
   dd if=/dev/zero of=/dev/null bs=1k count=2M
   cat /dev/zero | dd of=/dev/null bs=1k count=2M
look at the transfer rates reported and the system time.
-- 
Bill Davidsen <davidsen@xxxxxxx>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

-- 
users mailing list
users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines

[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [EPEL Devel]     [Fedora Magazine]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Desktop]     [Fedora Fonts]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Fedora Sparc]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux