On Mon, Dec 7, 2020 at 6:17 PM Mohamed Abuelfotoh, Hazem <abuehaze@xxxxxxxxxx> wrote: > > >Thanks for testing this, Eric. Would you be able to share the MTU > >config commands you used, and the tcpdump traces you get? I'm > >surprised that receive buffer autotuning would work for advmss of > >around 6500 or higher. > > Packet capture before applying the proposed patch > > https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-bad-unpatched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T170123Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=a599a0e0e6632a957e5619007ba5ce4f63c8e8535ea24470b7093fef440a8300 > > Packet capture after applying the proposed patch > > https://tcpautotuningpcaps.s3.eu-west-1.amazonaws.com/sender-bbr-good-patched.pcap?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJNMP5ZZ3I4FAQGAQ%2F20201207%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20201207T165831Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=f18ec7246107590e8ac35c24322af699e4c2a73d174067c51cf6b0a06bbbca77 > > kernel version & MTU and configuration from my receiver & sender is attached to this e-mail, please be aware that EC2 is doing MSS clamping so you need to configure MTU as 1500 on the sender side if you don’t have any MSS clamping between sender & receiver. > > Thank you. > > Hazem Please try again, with a fixed tcp_rmem[1] on receiver, taking into account bigger memory requirement for MTU 9000 Rationale : TCP should be ready to receive 10 full frames before autotuning takes place (these 10 MSS are typically in a single GRO packet) At 9000 MTU, one frame typically consumes 12KB (or 16KB on some arches/drivers) TCP uses a 50% factor rule, accounting 18000 bytes of kernel memory per MSS. -> echo "4096 180000 15728640" >/proc/sys/net/ipv4/tcp_rmem > > > On 07/12/2020, 16:34, "Neal Cardwell" <ncardwell@xxxxxxxxxx> wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On Mon, Dec 7, 2020 at 11:23 AM Eric Dumazet <edumazet@xxxxxxxxxx> wrote: > > > > On Mon, Dec 7, 2020 at 5:09 PM Mohamed Abuelfotoh, Hazem > > <abuehaze@xxxxxxxxxx> wrote: > > > > > > >Since I can not reproduce this problem with another NIC on x86, I > > > >really wonder if this is not an issue with ENA driver on PowerPC > > > >perhaps ? > > > > > > > > > I am able to reproduce it on x86 based EC2 instances using ENA or Xen netfront or Intel ixgbevf driver on the receiver so it's not specific to ENA, we were able to easily reproduce it between 2 VMs running in virtual box on the same physical host considering the environment requirements I mentioned in my first e-mail. > > > > > > What's the RTT between the sender & receiver in your reproduction? Are you using bbr on the sender side? > > > > > > 100ms RTT > > > > Which exact version of linux kernel are you using ? > > Thanks for testing this, Eric. Would you be able to share the MTU > config commands you used, and the tcpdump traces you get? I'm > surprised that receive buffer autotuning would work for advmss of > around 6500 or higher. > > thanks, > neal > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA SARL, Irish Branch, One Burlington Plaza, Burlington Road, Dublin 4, Ireland, branch registration number 908705 > >
