On Tue, Jun 18, 2024 at 6:56 AM Anton Gavriliuk <antosha20xx@xxxxxxxxx> wrote: > > Hi I have a NFS 4.2 server with 6x100 Gb/s ports. > > The NFS server connected via two switches to the ESXi host, 3 NFS > server ports per switch. > > Three VMs running with Oracle Linux 8.10 (kernel 4.18.0-553) > > On the VMs I mounted NFS share 6 times with 6 different ip addresses > > mount -o nconnect=16,max_connect=6,trunkdiscovery > 10.44.41.90:/exports/lv_prd6 /deep-storage2 > mount -o nconnect=16,max_connect=6,trunkdiscovery > 10.44.41.91:/exports/lv_prd6 /deep-storage2 > mount -o nconnect=16,max_connect=6,trunkdiscovery > 10.44.41.92:/exports/lv_prd6 /deep-storage2 > mount -o nconnect=16,max_connect=6,trunkdiscovery > 10.44.41.93:/exports/lv_prd6 /deep-storage2 > mount -o nconnect=16,max_connect=6,trunkdiscovery > 10.44.41.94:/exports/lv_prd6 /deep-storage2 > mount -o nconnect=16,max_connect=6,trunkdiscovery > 10.44.41.95:/exports/lv_prd6 /deep-storage2 > > During test I/O workload (8 fio jobs with different files per VM), > looking at two switches I see that single port shows much higher > bandwidth - > > 2 250 000 TX packets per second > > and rest of the traffic spreads across all other 5 ports evenly - > > 900 000 TX packets per second > > Please help determine why traffic doesn't spread evenly across all 6 > ports, more like 5+1 ? > > Seems like nconnect=16 works only for the 1st of 6 ip addresses. Nconnect only applies to the 1st (main) connection. In your example there should be 16 connections established to 10.44.41.90 and then 5 other trunked connections are going to be added to the group of the 16 nconnect connections. RPCs are going to be round robined over the 16+5 connections. 10.44.41.90 would see more traffic then the rest of IPs because it has more connections.
