Hi, dear community, In our NFS environment, NFS client mounts remote NFS export with its VIP. The VIP can be assigned to another server node for failover. However, the NFS client sends the unacknowledged packet 50s+ after the VIP is ready on the new node, which is because of the exponential backoff retransmission algorithm. I tried to set this parameter "tcp_retries2" smaller so that the NFS client can reconnect with the new node more quickly, but this parameter didn't take effect. From tcpdump entries as follows, 1. At "2024-05-29 11:47:00", ARP is updated. 2. At "2024-05-29 11:47:52" , the NFS client retried to send the packet. 3. Then the connection is reset and a new connection starts. I guess the parameter just takes effect for applications and doesn't take effect for kernel modules like the NFS client. Could anyone give some advice to customize retransmission timeout of unacknowledged NFS v3 TCP packet? OS: Linux kernel v6.7.0 NFS mount options: vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport tcp_retries2: [root@vm-play zhitaoli]# sysctl -w net.ipv4.tcp_retries2=5 net.ipv4.tcp_retries2 = 5 [root@vm-play zhitaoli]# cat /proc/sys/net/ipv4/tcp_retries2 5 tcpdump entries: 2024-05-29 11:46:02.331891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973659245 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:02.542836 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973659456 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:02.751013 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973659664 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:03.166958 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973660080 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:04.046882 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973660960 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:05.710910 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973662624 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:09.039310 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973665952 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:16.017889 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973672930 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:29.326891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973686240 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:46:55.950915 52:54:00:1d:a4:24 > 52:54:00:a0:93:93, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973712864 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:47:00.379844 52:54:00:13:1f:34 > Broadcast, ethertype ARP (0x0806), length 60: Reply 10.125.1.85 is-at 52:54:00:13:1f:34, length 46 2024-05-29 11:47:52.271192 52:54:00:1d:a4:24 > 52:54:00:13:1f:34, ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356, options [nop,nop,TS val 1973769184 ecr 28456 58566], length 124: NFS request xid 1954624602 120 access fh Unknown/43000001180100000000000000DE40020000000000F439000000000000000000 NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|NFS_ACCESS_DELETE 2024-05-29 11:47:52.272041 52:54:00:13:1f:34 > 52:54:00:1d:a4:24, ethertype IPv4 (0x0800), length 54: 10.125.1.85.nfs > 10.125.1.214.58428: Flags [R], seq 1148562527, win 0, length 0 2024-05-29 11:47:52.272909 52:54:00:1d:a4:24 > 52:54:00:13:1f:34, ethertype IPv4 (0x0800), length 74: 10.125.1.214.58428 > 10.125.1.85.nfs: Flags [S], seq 1734997801, win 32120, options [mss 1460,sackOK,TS val 1973769186 ecr 0,nop,wscale 7], length 0 2024-05-29 11:47:52.273503 52:54:00:13:1f:34 > 52:54:00:1d:a4:24, ethertype IPv4 (0x0800), length 74: 10.125.1.85.nfs > 10.125.1.214.58428: Flags [S.], seq 1078843840, ack 1734997802, win 28960, options [mss 1460,sackOK,TS val 2235915769 ecr 1973769186,nop,wscale 7], length 0 Best regards, Zhitao Li
