Hello team, I'd appreciate any help, suggestions or hints regarding my TCP RST issue with multipath routing.
I see random TCP connection resets, and while investigating I am stuck with understanding what part of the kernel can call the `ip_forward()` other than `ip_route_input_noref()`.
I have the following setup:
Kernel 6.4.0.
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.fib_multipath_hash_policy=1
```
```
# ip r
192.168.200.0/24 dev enp0s8 proto kernel scope link src 192.168.200.5 # network with backends
192.168.222.0/24 dev enp0s9 proto kernel scope link src 192.168.222.5 # client network
10.100.100.0/24 proto bird # VIP network
nexthop via 192.168.200.101 dev enp0s8 weight 1
nexthop via 192.168.200.102 dev enp0s8 weight 1
```
```
192.168.222.99 - client IP
192.168.200.101 and 192.168.200.102 - real IPs of 2 backends.
10.100.100.1 - a VIP, backends have it on lo dev.
```
Backends and a client have the router as a default gateway.
Example of a RST with my investigations are the following.
Using `bpftrace` I can see that `ip_route_input_noref()` is run 3 times before the connection reset:
```
ack_seq: 0, 192.168.222.99:46308 -> 10.100.100.1:8080, 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff
ack_seq: 322960818, 192.168.222.99:46308 -> 10.100.100.1:8080, 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff
ack_seq: 0, 192.168.222.99:46308 -> 10.100.100.1:8080, 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff
```
but `ip_forward()` runs 4 times with a wrong `gw` in the `skb->_skb_refdst` at the middle of a connection. It should be 192.168.200.101, but it uses 192.168.200.102 instead:
```
ack_seq: 0, 192.168.222.99:46308 -> 10.100.100.1:8080, 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff, gw: 192.168.200.101
ack_seq: 322960818, 192.168.222.99:46308 -> 10.100.100.1:8080, 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff, gw: 192.168.200.101
ack_seq: 322960827, 192.168.222.99:46308 -> 10.100.100.1:8080, 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff, gw: 192.168.200.102
ack_seq: 0, 192.168.222.99:46308 -> 10.100.100.1:8080, 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff, gw: 192.168.200.101
```
I locally added logging to `__mkroute_input()` to understand if it's a hashing problem, but it also shows me only 3 calls and a correct `rt_gw4` in `rth rtable` and `nhc`:
```
192.168.222.99:46308 -> 10.100.100.1:8080, mac: 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff, gw4: 192.168.200.101, rt_gw4:192.168.200.101
192.168.222.99:46308 -> 10.100.100.1:8080, mac: 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff, gw4: 192.168.200.101, rt_gw4:192.168.200.101
192.168.222.99:46308 -> 10.100.100.1:8080, mac: 8:0:27:a8:29:e5 -> 8:0:27:86:65:ff, gw4: 192.168.200.101, rt_gw4:192.168.200.101
````
I am stuck with understanding how it's possible that the kernel makes an additional incorrect `ip_forward()` call, when neither `__mkroute_input()` nor `ip_route_input_noref()` calls it.
Thank you for any hints.
--
Viacheslav Biriukov
BR
BR
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@xxxxxxxxxxxxxxxxx https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
