Linux is notoriously slow for large copies over usb3. The reading I've done on it seems to suggest that it has to do with how buffering in RAM works. There doesn't seem to be an elegant solution. Some solutions posted are: 1) Make sure the external drive is ext4 or other linux-friendly format, rather than ntfs. Linux support for ntfs is inefficient, apparently. This helped significantly for me, but it didn't solve the issue. 2) For non-system-wide backups, tar and then dd with a reasonable bs option. This also helped, but I kept running out of disk space for large tar files. 3) Play with dirty bytes configuration (see: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/500069 ). I haven't tested this. 4) A rather blunt solution that helped for me for backing up my home directory, was to break up the backup into chunks and let the buffer empty as it moved from command to command e.g. rsync -avs /home/$user/[aA]* /mnt/hd/$backupdir rsync -avs /home/$user/[bB]* /mnt/hd/$backupdir rsync -avs /home/$user/[cC]* /mnt/hd/$backupdir . . . rync -avs /home/$user/.* /mnt/hd/$backupdir I didn't think it would help, but it did. Obviously, that's not a good solution for system wide backups, but it speeds up home directory and project directory backups for me. I couldn't find an option for rsync to pause and clear buffer after every 25 GB or so... billo On Mon, 2023-10-30 at 12:20 +0000, Patrick O'Callaghan wrote: > On Mon, 2023-10-30 at 22:58 +1100, fedora@xxxxxxxxxxxxxx wrote: > > On 30/10/2023 22.41, Patrick O'Callaghan wrote: > > > On Mon, 2023-10-30 at 22:35 +1100, fedora@xxxxxxxxxxxxxx wrote: > > > > On 30/10/2023 21.40, Patrick O'Callaghan wrote: > > > > > On Mon, 2023-10-30 at 18:22 +1100, > > > > > fedora@xxxxxxxxxxxxxx wrote: > > > > > > Should I blame rsync (or the way I use it)? > > > > > > > > > > > > Next, I will do a plain 'cp -a' between the original source > > > > > > (sdh > > > > > > on > > > > > > /sata) and target(md127 on /data1) to see how it goes. > > > > > > > > > > If you say how you use it, someone might be able to answer > > > > > that. > > > > > Give > > > > > the exact command line with all options (e.g. if you have > > > > > strict > > > > > change > > > > > detection you might be checksumming every file on the source > > > > > before > > > > > deciding to copy it). > > > > > > > > First, I already did this test and it still chokes after a few > > > > GB. > > > > The problem is still there > > > > and it is not only an rsync problem. But why does rsync use > > > > 100% > > > > CPU? > > > > > > > > I thought that I gave the command but I did not initially, yet > > > > when I > > > > show the output of 'ps' it says: > > > > sudo rsync -aHSK --stats --progress --checksum- > > > > choice=none -- > > > > no-compress -W /sata/backups/tapes /data1/no-backup/old- > > > > backups/ > > > > so there it is. > > > > > > Nothing obvious at first glance. The '-W' option should mean that > > > rsync > > > is simply copying files without it's usual delta detection, which > > > could > > > have been the culprit. > > > > The culprit of rsync being slow. > > > I lost you here. 'the culprit' of what? It was very slow initially. > > I then added '-W' hoping to speed it up but it did not help. > > > > I also added '--checksum-choice=none --no-compress' for the same > > reason (probably > > not needed) but none had an effect. > > > > That's what I assumed. > > > I suspect this may be related to another thread I started earlier > > 'kworker consumes 100% CPU on degraded RAID6' > > Possibly. A good test would be to repeat the experiment but with a > different target destination, such as /dev/null. This would indicate > whether the problem is with reading or with writing. > > poc > _______________________________________________ > users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue _______________________________________________ users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
