Re: Parallel transfers with sftp (call for testing / advice)

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

 



> Le 10 avr. 2020 à 01:55, Darren Tucker <dtucker@xxxxxxxxxxx> a écrit :
> 
> On Thu, 9 Apr 2020 at 01:34, Cyril Servant <cyril.servant@xxxxxxxxx> wrote:
> [...]
>> Each of our front
>> nodes has an outgoing bandwidth limit (let's say 1Gb/s each, generally more
>> limited by the CPU than by the network bandwidth),
> 
> You might also want to experiment with the Ciphers and MACs since
> these can make a significant difference in CPU utilization and, if
> that's the bottleneck, your throughput.  Which one is best will vary
> depending on your hardware, but it's likely to be either AES GCM if
> the hardware has AES instructions or chacha20-poly1305 if not.
> 
> In the first example below the bottleneck is the source's relatively
> elderly 2.66GHz Intel CPU.  In the second it's the gigabit network
> between them.
> 
> $ scp -c aes256-ctr -o macs=hmac-sha2-512
> ubuntu-18.10-desktop-amd64.iso.bz2 nuc:/tmp/
> ubuntu-18.10-desktop-amd64.iso.bz2            100% 1899MB  63.5MB/s   00:29
> 
> $ scp -c chacha20-poly1305@xxxxxxxxxxx
> ubuntu-18.10-desktop-amd64.iso.bz2 nuc:/tmp/
> ubuntu-18.10-desktop-amd64.iso.bz2            100% 1899MB 112.1MB/s   00:16

Yes, we already optimised the ciphers and macs used, and indeed recent CPUs
are pretty damn fast with some of them. But parallelising transfers gives us
one more order of magnitude concerning transfer speed.

[me@france openssh-portable]$ sftp germany
Connected to germany.
sftp> get /files/5g 5g.bis
Fetching /files/5g to 5g.bis
/files/5g                                 100% 5120MB 84.5MB/s   01:00
sftp> put 5g /files/5g
Uploading 5g to /files/5g
5g                                        100% 5120MB 81.4MB/s   01:02

[me@france openssh-portable]$ ./sftp -n 10 germany
Connected main channel to germany (1.2.3.104).
Connected channel 1 to germany (1.2.3.102).
Connected channel 2 to germany (1.2.3.105).
Connected channel 3 to germany (1.2.3.103).
Connected channel 4 to germany (1.2.3.96).
Connected channel 5 to germany (1.2.3.98).
Connected channel 6 to germany (1.2.3.101).
Connected channel 7 to germany (1.2.3.97).
Connected channel 8 to germany (1.2.3.100).
Connected channel 9 to germany (1.2.3.99).
Connected channel 10 to germany (1.2.3.104).
sftp> get /files/5g 5g.bis
Fetching /files/5g to 5g.bis
/files/5g                                100% 5120MB 748.8MB/s   00:06
sftp>  put 5g /files/5g
Uploading 5g to /files/5g
5g                                       100% 5120MB 697.1MB/s   00:07

-- 
Cyril
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@xxxxxxxxxxx
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev




[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux