Using NAT engine information to apply fwmark to packet

["Tobias Jakobi (Compleo)" <tobias.jakobi.compleo@xxxxxxxxx>]

Hello everyone,

looking for some pointers concerning this problem. So what I'm trying to 
achieve is to base the routing decision of a packet on the question "Was 
this packet handled by the NAT engine?".

I'm using systemd-networkd to setup NAT / masquerading on the system, 
which basically means that networkd enables forwarding for the correct 
interfaces and adds a nftables table.

Which looks like this:
table ip io.systemd.nat {
    set masq_saddr {
        type ipv4_addr
        flags interval
        elements = { 192.168.15.0/24 }
    }

    map map_port_ipport {
        type inet_proto . inet_service : ipv4_addr . inet_service
    }

    chain prerouting {
        type nat hook prerouting priority dstnat + 1; policy accept;
        fib daddr type local dnat ip to meta l4proto . th dport map 
@map_port_ipport
    }

    chain output {
        type nat hook output priority -99; policy accept;
        ip daddr != 127.0.0.0/8 oif "lo" dnat ip to meta l4proto . th 
dport map @map_port_ipport
    }

    chain postrouting {
        type nat hook postrouting priority srcnat + 1; policy accept;
        ip saddr @masq_saddr masquerade
    }
}

My naive approach was to add another table, which looks like this:
table ip nat.test {
    chain prerouting {
        type nat hook prerouting priority dstnat + 2; policy accept;
        meta mark set 0x00000007
    }
}

I can then use the fwmark 0x7 in my routing policy rule to select an 
appropriate routing table.

However this does not work properly.

When I issue a simple ping to 8.8.8.8 from my NAT client (which has the 
192.168.15.2), then I see that the ICMP packet pops up on the gateway 
(the 192.168.15.1), is then masqueraded and tagged with the fwmark, 
before eventually being send to the Google DNS, where it is answered.

So this direction "NAT client -> outside world" seems to be working. 
What is not working is the when the Google DNS replies and then whole 
masquerading stuff is done in reverse. In this case I don't see any 
fwmark being added, and hence the routing policy rule does not apply.

Looking at the netfilter flowchart 
(https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks) I 
get the impression that applying rules with priority dstnat + 2 is 
probably too late when the NAT engine detects that a packet belongs to a 
connection that it is tracking.

So I guess my question is: I this whole thing even possible, and if yes, 
how do I extract the required information from the NAT engine?

I'm currently looking into netfilter's connmark functionality, as the 
NAT engine is basically conntrack?! (is it really?) And it seems to 
allow to transfer connection marks with packet marks, which I probably 
need here.

Thanks in advance!

With best wishes,
Tobias