David Woodhouse wrote on 11/13/18 2:01 PM: > On Tue, 2018-11-13 at 13:56 -0800, Tom Rodriguez wrote: >> Thanks, I wasn't aware of that command. That does indeed fix the >> throughput problem on linux though I'll have to rework some things to >> take advantage of it. Unfortunately it's not available for the Mac. I >> can work with it though. Having a partial solution is better than none. >> >> So that suggests that the bottleneck is really in ocproxy. Clearly >> openconnect can communicate with the tunnel process efficiently enough >> to support the throughput. I've investigated ocproxy before to see if >> there was something wrong with the LWIP configuration that was limiting >> throughput but didn't see anything. Anyway, thanks for the vpnns tip. > > Does LWIP support SACK yet? Are you seeing packet loss? Do you see SACK > being used when you use vpnns and the kernel's stack? ocproxy usesfff lwip 1.4.0 which doesn't support SACK, though the latest version does. I ported it to use the latest lwip in hopes that would solve my problem but it made no difference to the throughput. I did look at the tcpdump style output ocproxy -T produces and there appear be some problems with packet loss and retransmission but that only occurs when I'm not getting full throughput. I've included a snippet of the log at the end of this message. The normal pattern is 2 data packets and 1 ack but sometimes it seem to miss every other packet for a while and just keeps acking the last received packet until it eventually resyncs. I think this explains the occasional very bad throughput I see with ocproxy. The openconnect verbose output doesn't seem to indicate that it's the one dropping the packets so they must be real drops I guess. Anyway the -T output of my best throughput with ocproxy looks pretty much the same as the tcpdump of the fast throughput with vpnns. The wnd and message sizes are slightly different. > > Or maybe it's just TCP congestion algorithms and window sizing... > Yeah I'm will to accept that it might that lwip just doesn't cope that well the stream it's seeing but I assume it can do better than 1.5MB/s. tom This is a ocproxy -T dump of a bad retransmission segment during a transfer that was only getting about 700KB/s instead of the normal 1.5MB/s. 10.159.148.137.64242 > 10.213.24.141.80: ack 3592752147 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592752147:3592753395(1248) ack 6734 wnd 30016 10.213.24.141.80 > 10.159.148.137.64242: . 3592753395:3592754643(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592754643 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592754643:3592755891(1248) ack 6734 wnd 30016 10.213.24.141.80 > 10.159.148.137.64242: . 3592755891:3592757139(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592758387:3592759635(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592759635:3592760883(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592760883:3592762131(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592762131:3592763379(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592763379:3592764627(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592764627:3592765875(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592765875:3592767123(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592767123:3592768371(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592768371:3592769619(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592769619:3592770867(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592770867:3592772115(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592772115:3592773363(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592773363:3592774611(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592774611:3592775859(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592775859:3592777107(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592777107:3592778355(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592778355:3592779603(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592779603:3592780851(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592780851:3592782099(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592782099:3592783347(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592783347:3592784595(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592784595:3592785843(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592785843:3592787091(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592787091:3592788339(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592788339:3592789587(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592789587:3592790835(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592790835:3592792083(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592792083:3592793331(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592793331:3592794579(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592794579:3592795827(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592795827:3592797075(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592797075:3592798323(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592798323:3592799571(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592799571:3592800819(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592800819:3592802067(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592802067:3592803315(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592803315:3592804563(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592804563:3592805811(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592805811:3592807059(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592807059:3592808307(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592808307:3592809555(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592809555:3592810803(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592810803:3592812051(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592757139 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592757139:3592758387(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592812051 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592812051:3592813299(1248) ack 6734 wnd 30016 10.213.24.141.80 > 10.159.148.137.64242: . 3592813299:3592814547(1248) ack 6734 wnd 30016 10.159.148.137.64242 > 10.213.24.141.80: ack 3592814547 wnd 65534 10.213.24.141.80 > 10.159.148.137.64242: . 3592814547:3592815795(1248) ack 6734 wnd 30016 10.213.24.141.80 > 10.159.148.137.64242: . 3592815795:3592817043(1248) ack 6734 wnd 30016
