nftables DNAT routes to wrong iface

[Marc SCHAEFER <schaefer@xxxxxxxxxxx>]

Hello,

I am trying to do this:

Both local IP addresses, when access from outside:

   enp2s0.310: 193.72.186.130:8080
   br0: 46.140.72.218:8080

are DNATed to

   enp2s0.202: 192.168.202.10:80 (a remote machine)

I know there are two steps:

   - the incoming DNAT in prerouting (and the forward accept)
     -> this part I am stuck, see below

   - then when the reply comes back, route to the proper interface where
     it came from (using conntrack + marks + specific routing tables)
     -> this part I have not done yet -- it would be required for
        193.72.186.130:8080 obviously because the default route
        does not go there.

What I observe:

   telnet 46.140.72.218 8080 from outside works (*), it connects to 192.168.202.10:80
   and there is nothing bizarre in tcpdump either on br0 nor on
   enp2s0.202 (no delays, lost packets, e.g.) (**)

   telnet 193.72.186.130 8080 gives this on enp2s0.300

      IP 193.72.186.190.52636 > 193.72.186.130.8080 (normal)
      IP 193.72.186.190.52636 > 192.168.202.10.80   (good, it was DNATted, BUT should be on enp2s0.202!)

   aka the DNAT is executed, but then 192.168.202.10 is not routed
   correctly.  From the diagram https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks
   I thought that routing would be done AFTER prerouting/DNAT.

The routing table has:

default via 46.140.72.217 dev br0 onlink 
46.140.72.216/29 dev br0 proto kernel scope link src 46.140.72.218 
192.168.202.0/24 dev enp2s0.202 proto kernel scope link src 192.168.202.2 
193.72.186.128/26 dev enp2s0.300 proto kernel scope link src 193.72.186.130 

(*) not from 193.72.186.0/24, however, gets the same bug.

The nftables config:

table ip filter {
        # obviously a later goal is also to encode the L4 protocol here
        # and not hardcode it in the prerouting
        map multihoming_ext {
                type ipv4_addr . inet_service : ipv4_addr . inet_service
                elements = { 193.72.186.130 . 8080 : 192.168.202.10 . 80,
                             46.140.72.218 . 8080 : 192.168.202.10 . 80 }
        }

        set w_all {
                type ipv4_addr
                flags interval
                elements = { 46.140.72.216/29,
                             192.168.202.1, 193.72.186.0/24
                           }
        }

        chain input {
                type filter hook input priority filter; policy drop;
                ct state invalid counter packets 3 bytes 120 drop
                ct state { established, related } counter packets 1092 bytes 76846 accept
                iif "lo" counter packets 0 bytes 0 accept
                counter packets 324 bytes 13539 jump whitelist
                counter packets 323 bytes 13455 jump blacknets
                counter packets 323 bytes 13455 jump blacklist
                counter packets 323 bytes 13455 jump incoming
        }

        chain forward {
                type filter hook forward priority filter; policy drop;
                ct state invalid counter packets 0 bytes 0 drop
                ct state { established, related } counter packets 12 bytes 548 accept

                # already after DNAT, obviously
                iifname "br0" ip daddr 192.168.202.10 tcp dport 80 accept
                iifname "enp2s0.300" ip daddr 192.168.202.10 tcp dport 80 accept
        }

        chain output {
                type filter hook output priority filter; policy accept;
                ct state invalid counter packets 0 bytes 0 drop
                ct state { established, related } counter packets 890 bytes 182096 accept
                oif "lo" counter packets 0 bytes 0 accept
                counter packets 1 bytes 76 jump outgoing
        }

        chain rejectcounter {
                meta l4proto tcp counter packets 0 bytes 0 reject with tcp reset
                meta l4proto udp counter packets 0 bytes 0 reject
                counter packets 0 bytes 0 drop
        }

        chain dropcounter {
                counter packets 320 bytes 13349 drop
        }

        chain whitelist {
                ip saddr @w_all counter packets 1 bytes 84 accept
        }

        chain blacknets {
        }

        chain blacklist {
        }

        chain incoming {
                icmp type echo-request counter packets 3 bytes 106 accept
                icmp type echo-reply counter packets 0 bytes 0 accept
                tcp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
                udp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
                counter packets 320 bytes 13349 jump dropcounter
        }

        chain outgoing {
        }

        chain multihoming_prerouting {
                type nat hook prerouting priority dstnat; policy accept;

                # this is the DNAT
                dnat ip to ip daddr . tcp dport map @multihoming_ext
        }
}
table ip myhelpers {
        chain prerouting {
                type filter hook prerouting priority filter; policy accept;
        }
}

[ I removed ip6 entries ]

Any idea what could be wrong?

Thank you.

(**) very nice:
br0:
IP 46.140.72.222.60394 > 46.140.72.218.8080
IP 46.140.72.218.8080 > 46.140.72.222.60394
IP 46.140.72.222.60394 > 46.140.72.218.8080

enp2s0.202 (another connection):
IP 46.140.72.222.57790 > 192.168.202.10.80
IP 192.168.202.10.80 > 46.140.72.222.57790
IP 46.140.72.222.57790 > 192.168.202.10.80