Cyril Servant wrote: > >> 2. The solution > >> > >> I made some changes in the sftp client. The new option "-n" (defaults to 0) sets > >> the number of extra channels. There is one main ssh channel, and n extra > >> channels. The main ssh channel does everything, except the put and get commands. > >> Put and get commands are parallelized on the n extra channels. Thanks to this, > >> when the customer uses "-n 5", he can transfer his files up to 5Gb/s. There is > >> no server side change. Everything is made on the client side. ... > I can fully understand this. In our case, the network is not really crowded, as > customers are generally using research / educational links. Indeed, this is > totally a niche case, but still a need for us. The main use case is putting data > you want to process into the cluster, and when the job is finished, getting the > output of the process. There is rarely the need for synchronising files, except > for the code you want to execute on the cluster, which is considered small > compared to the data. rsync is the obvious choice for synchronising the code, > but not for putting / getting huge amounts of data. When I read through the thread immediately I thought this would have a good chance of speeding the transfer up by simulating sharing of the bandwidth through unfair sharing. Which may still be fair. But that was my thought. With some simplification for discussion... A network connection that is handling multiple connections will share the bandwidth among those connections. Let's assume every connection is using whatever the maximum packet size is for that link and not worry about efficiency of different sized packets on the link. The effect is that if two connections are streaming then each will get half of the available bandwidth. And if 10 are streaming then each will get 1/10th of the bandwidth. And if 100 then each will get 1/100th. Which means that *if* a connection is mostly operating near'ish capacity then a user can get more of it by using more parallel connections. As scheduling walks through the connections giving each bandwidth in turn then the one with more parallel connections will get more of it. Let's assume a link with 20 connections streaming. If I add my own then I am number 21 and everyone will get 1/21 of the bandwidth. But if instead I open 50 connections and transfer in parallel then 20+50=70 and each connection getting 1/70th of the bandwidth. But since 50 of those are mine I get 50/70 of the bandwidth for my own data and each of the other 20 users get 1/70 for those other 20 users. I say this based upon experience with software that relied upon this behavior transferring big data in my previous career. It's been a few years but we had the same issue and a very similar solution back then. Someone hacked together a process to chop up big data files and transparently send them in parallel assembling them on the remote end. At the time for our case it was leased lines between sites. Using this technique one group was able to get high priority data transferred between sites faster. Faster by slowing down the other people in the company who were also transferring data across the same network links. But this was good because the high priority data had priority. Whether that is good or bad depends upon all of the details. If the connection is my network, or my group's network, and this is the priority task, then this is a good way to make use of more bandwidth. Think of it like Quality of Service tuning. If on the other hand it is me borrowing someone else's network and I am negatively impacting them, then maybe not so good. For example if I were downloading data in my home neighborhood and preventing my neighbors from streaming video during this time of "stay-at-home let's defeat the COVID-19 virus" then that would be a bad and unfair use. It's neither good nor bad intrinsically but just how it is used that is important. I am thinking that this improves bandwidth for one connection by implementing parallel connections which allows a greater share of the bandwidth. I might be completely wrong here. It's just my thought from sitting back in my comfy chair while staying at home to defeat the virus. And you asked for any thoughts... Bob _______________________________________________ openssh-unix-dev mailing list openssh-unix-dev@xxxxxxxxxxx https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
