More info. One of my coworkers and I did some more diagnostics this afternoon: He used iperf3 to do some generic benchmarking today, and we poked around in some tcpdumps taken during 1MB iperf pushes. When sending TCP traffic from the open connect client VM to the target (an ordinary RHEL VM, recall that the VPN is terminated on a hardware Palo Alto along the way) we observe an average of about 2.7MB/sec with around 10% of packets getting retransmitted according to both iperf and netstat -s. The first packet that gets retransmitted after the TCP socket is opened has relative sequence number 4142 consistently... as in every time we tried (at least four separate runs during an hour or so). When going in the other direction, we get 36MB/sec with about 0.3% of packets getting retransmitted. We didn’t do as much poking in these dumps these since they performed 10x-15x faster, but I would hazard a bet that in these cases, it’s the ACKs that are getting lost dropped. This definitely hints (if not shouts) at a send buffer overflow on the VPN side. So, your observation below references adjusting TUNSETSNDBUF. Can you give me a quick how-to to do this, and what changes would be “sane?’ > On Mar 18, 2019, at 4:22 PM, Phillips, Tony <tonyphillips@xxxxxx> wrote: > > That's a great question! The answer seems to be "Neither" (at least, using commands with which I'm familiar). > > These commands were done on a freshly booted VM where a few data movement tests were performed before starting the below. > > > # netstat -i > Kernel Interface table > Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg > eth0 1500 82560 0 0 0 86307 0 0 0 BMRU > lo 65536 278 0 0 0 278 0 0 0 LRU > tun0 1422 82192 0 0 0 89595 0 0 0 MOPRU > > But: > > # netstat -s | egrep -i 'drop|lost|retrans' > 3343 outgoing packets dropped > 101 dropped because of missing route > 2975 segments retransmited > TCPLostRetransmit: 5 > 2963 fast retransmits > 2 forward retransmits > 3 retransmits in slow start > 1 SACK retransmits failed > > And, after writing a ~30meg file: > > 4731 outgoing packets dropped > 101 dropped because of missing route > 4362 segments retransmited > TCPLostRetransmit: 5 > 4349 fast retransmits > 2 forward retransmits > 3 retransmits in slow start > 1 SACK retransmits failed > TCPSynRetrans: 1 > > # netstat -i > Kernel Interface table > Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg > eth0 1500 92832 0 0 0 111539 0 0 0 BMRU > lo 65536 308 0 0 0 308 0 0 0 LRU > tun0 1422 92391 0 0 0 116211 0 0 0 MOPRU > > ...and if I set on the machine and just "look around" reading man pages or whatever for 10 minutes, the netstat -s results remain static. > > So it appears that things go wonky only when under stress. > > As far as getting access to a test environment, it's possible, but will take a lot of coordination on our end so would like to hold that as a last resort for now. > > > > -----Original Message----- > From: David Woodhouse [mailto:dwmw2@xxxxxxxxxxxxx] > Sent: Sunday, March 17, 2019 4:39 PM > To: Nikos Mavrogiannopoulos; Phillips, Tony > Cc: openconnect-devel@xxxxxxxxxxxxxxxxxxx > Subject: Re: [EXTERNAL] Re: What throughput is reasonable? > >> On Sun, 2019-03-17 at 12:44 +0000, Nikos Mavrogiannopoulos wrote: >> David, Could it be we set a smaller snd or recv buffer to reduce >> latency in openconnect? In ocserv we have something configurable for >> that. BTW Tony, have you tried a more recent rhel7? > > Tony, on which interface are the packets being dropped? Is it the tun > device? If so, perhaps increasing the buffer with TUNSETSNDBUF might > help? > > Are you able to give us access to a test environment? > _______________________________________________ openconnect-devel mailing list openconnect-devel@xxxxxxxxxxxxxxxxxxx http://lists.infradead.org/mailman/listinfo/openconnect-devel
