Re: SCP with Resume Feature

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

 



On Tue, 6 Apr 2021, rapier wrote:

> On 4/6/21 10:04 PM, Damien Miller wrote:
> > On Tue, 6 Apr 2021, rapier wrote:
> > 
> > > Looking at the performance - on my systems sftp seems to be a bit slower
> > > than scp when dealing with a lot of small files. Not sure why this is
> > > the case as I haven't looked at the sftp code in years.
> > 
> > the OpenSSH sftp client doesn't do inter-file pipelining - it only
> > pipelines read/writes within a transfer, so each new file causes a
> > stall.
> > 
> > This is all completely fixable on the client side, and shouldn't apply
> > to things like sshfs at all.
> 
> Gotcha. Is this because of how it sequentially loops through the readdirs in
> two _dir_internal functions?

Only partly - the client will do SSH2_FXP_READDIR to get the full list of
files and then transfer each file separately. The SSH2_FXP_READDIR are not
pipelined at all, there is no pipelining between obtaining the file list
and the file transfers. Finally each file transfer incurrs a pipeline
stall upon completion.

> If so I'm wondering if you could spawn per file
> threads to get some concurrency within a directory.

I don't think we want a threaded sftp client and AFAIK it isn't necessary
for the main problem. We could add inter-operation pipelining by adding a
work queue structure and driving the next operation from that. This is
similar to what happens inside do_download()/do_upload() already, but
extended to persist across and between different operations.

> Just curious and this is
> the first time I've looked at the sftp code in years. I hope you don't mind
> the questions.

Not at all :)

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@xxxxxxxxxxx
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev



[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux