On Tue, 25 Jan 2022 at 22:11, Patrick Goetz <pgoetz@xxxxxxxxxxxxxxx> wrote: > > IDK, 4000 images per collection, with hundreds of collections on disk? > Say at least 500,000 files? Maybe a million? With most files about 1GB > in size. I was trying to just rsync it all from the data server to a > ZFS-based backup server in our data center, but the backup started > failing constantly because the filesystem would change after rsync had > already constructed an index. Even after an initial copy, a backup like > that runs for over a week. The strategy I'm about to try and implement > is to NFS mount the data server's data partition to the backup server > and then have a script walk through the directory hierarchy, rsyncing > collections one at a time. ZFS send/receive would probably be better, > but the data server isn't configured with ZFS. We've strayed slightly off topic (even if we are talking about file creates over NFS) because you can get good parallel performance (creates, read, writes etc) over NFS with simultaneous copies using lots of processes if distributed across lots of directories. Well "good" being subjective. I get 1,500 creates/s in a single directory on a LAN NFS server from a single client and 160 creates/s aggregate over my extreme 200ms using 10 clients & 10 different directories. It seems fair all things considered. But seeing as I do a lot of these kinds of big data moves (TBs) across both the LAN and WAN, I can perhaps offer some advice from experience that might be useful: * walk the filesystem (locally) first to build a file list, split it and then use rsync --files-from (e.g. https://github.com/jbd/msrsync) to feed multiple simultaneous rsyncs. * avoid NFS and use rsyncd directly between the servers (no ssh) so filesystem walks are "local". The advantage of rsync is that it will do the filesystem walks at both ends locally and compare the directory trees as it goes along. The other nice thing it does is open a connection between sender and receiver and stream all the file data down it so it works really well even for lists of small files. The TCP connection and window scaling can sit at it's maximum without any slow remote file metadata latency disrupting it. Avoid the encapsulation of sshand use rsyncd instead as it just speeds everything up. And as always with any WAN connection, large buffers, window scaling, no firewall DPI and maybe some fancy congestion control like BBR/2 helps. Daire
