From: Eric Dumazet <edumazet@xxxxxxxxxx> Date: Fri, 26 Aug 2022 08:17:25 -0700 > On Thu, Aug 25, 2022 at 5:05 PM Kuniyuki Iwashima <kuniyu@xxxxxxxxxx> wrote: > > > > The more sockets we have in the hash table, the more time we spend > > looking up the socket. While running a number of small workloads on > > the same host, they penalise each other and cause performance degradation. > > > > Also, the root cause might be a single workload that consumes much more > > resources than the others. It often happens on a cloud service where > > different workloads share the same computing resource. > > > > On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash > > entries), after running iperf3 in different netns, creating 24Mi sockets > > without data transfer in the root netns causes about 10% performance > > regression for the iperf3's connection. > > > > thash_entries sockets length Gbps > > 524288 1 1 50.7 > > 24Mi 48 45.1 > > > > It is basically related to the length of the list of each hash bucket. > > For testing purposes to see how performance drops along the length, > > I set 131072 (1Mi / 8) to thash_entries, and here's the result. > > > > thash_entries sockets length Gbps > > 131072 1 1 50.7 > > 1Mi 8 49.9 > > 2Mi 16 48.9 > > 4Mi 32 47.3 > > 8Mi 64 44.6 > > 16Mi 128 40.6 > > 24Mi 192 36.3 > > 32Mi 256 32.5 > > 40Mi 320 27.0 > > 48Mi 384 25.0 > > > > To resolve the socket lookup degradation, we introduce an optional > > per-netns hash table for TCP and UDP. With a smaller hash table, we > > can look up sockets faster and isolate noisy neighbours. Also, we can > > reduce lock contention. > > > > We can control and check the hash size via sysctl knobs. It requires > > some tuning based on workloads, so the per-netns hash table is disabled > > by default. > > > > # dmesg | cut -d ' ' -f 5- | grep "established hash" > > TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage) > > > > # sysctl net.ipv4.tcp_ehash_entries > > net.ipv4.tcp_ehash_entries = 524288 # can be changed by thash_entries > > > > # sysctl net.ipv4.tcp_child_ehash_entries > > net.ipv4.tcp_child_ehash_entries = 0 # disabled by default > > > > # ip netns add test1 > > # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries > > net.ipv4.tcp_ehash_entries = -524288 # share the global ehash > > > > # sysctl -w net.ipv4.tcp_child_ehash_entries=100 > > net.ipv4.tcp_child_ehash_entries = 100 > > > > # sysctl net.ipv4.tcp_child_ehash_entries > > net.ipv4.tcp_child_ehash_entries = 128 # rounded up to 2^n > > > > # ip netns add test2 > > # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries > > net.ipv4.tcp_ehash_entries = 128 # own per-netns ehash > > > > [ UDP has the same interface as udp_hash_entries and > > udp_child_hash_entries. ] > > > > When creating per-netns concurrently with different sizes, we can > > guarantee the size by doing one of these ways. > > > > 1) Share the global hash table and create per-netns one > > > > First, unshare() with tcp_child_ehash_entries==0. It creates dedicated > > netns sysctl knobs where we can safely change tcp_child_ehash_entries > > and clone()/unshare() to create a per-netns hash table. > > > > 2) Lock the sysctl knob > > > > This is orthogonal. > > Your series should have been split in three really. > > I do not want to discuss the merit of re-instating LOCK_MAND :/ I see. I'll drop the flock() part at once and respin TCP part only in v2.
