Hello Phil,
It almost worked ( Check this out:
sudo nft list table ipv4table
table ip ipv4table {
set no-endpoint-svc-ports {
type inet_service
elements = { 8080, 8989 }
}
set no-endpoint-svc-addrs {
type ipv4_addr
flags interval
elements = { 10.1.1.1, 10.1.1.2}
}
chain input-net {
type nat hook input priority filter; policy accept;
jump services
}
chain input-local {
type nat hook output priority filter; policy accept;
jump services
}
chain services {
ip daddr @no-endpoint-svc-addrs tcp dport @no-endpoint-svc-ports reject with tcp reset
ip daddr @no-endpoint-svc-addrs udp dport @no-endpoint-svc-ports reject with icmp type net-unreachable
}
chain svc1-endpoint-1 {
ip protocol tcp dnat to 12.1.1.1:8080
}
chain svc1-endpoint-2 {
ip protocol tcp dnat to 12.1.1.2:8080
}
chain svc2-endpoint-1 {
ip protocol tcp dnat to 12.1.1.3:8090
}
chain svc2-endpoint-2 {
ip protocol tcp dnat to 12.1.1.4:8090
}
chain svc1 {
}
chain svc2 {
}
chain prerouting {
type nat hook prerouting priority filter; policy accept;
ip daddr 1.1.1.1 tcp dport 88 numgen random mod 2 vmap { 0 : jump svc1-endpoint-1, 1 : jump svc1-endpoint-2 }
ip daddr 2.2.2.2 tcp dport 99 numgen random mod 2 vmap { 0 : jump svc2-endpoint-1, 1 : jump svc2-endpoint-2 }
}}
Ideally I need to apply this rule " numgen random mod 2 vmap { 0 : jump svc1-endpoint-1, 1 : jump svc1-endpoint-2 }" to svc1 and svc2 chains to load balance between services' endpoints but when I do that it fails with Unsupported operation.
In contrast it let me apply this rule to prerouting chain.
This split support of reject in input/forward/output and numgen only in prerouting is not ideal as a packet for a client of a service without registered endpoint will need to go through all checks in prerouting chain before it reaches input chain and get its reject back.
Thank you very much for your help.
Serguei
On 2019-11-26, 2:28 PM, "n0-1@xxxxxxxxxxxxx on behalf of Phil Sutter" <n0-1@xxxxxxxxxxxxx on behalf of phil@xxxxxx> wrote:
Hi,
On Tue, Nov 26, 2019 at 06:47:09PM +0000, Serguei Bezverkhi (sbezverk) wrote:
> Ok, I guess I will work around by using input and output chain types, even though it will raise some brows in k8s networking community.
>
> I have a second issue I am struggling to solve with nftables. Here is a service exposed for tcp port 80 which has 2 corresponding backends listening on a container port 8080.
>
> !
> ! Backend 1
> !
> -A KUBE-SEP-FS3FUULGZPVD4VYB -s 57.112.0.247/32 -j KUBE-MARK-MASQ
> -A KUBE-SEP-FS3FUULGZPVD4VYB -p tcp -m tcp -j DNAT --to-destination 57.112.0.247:8080
> !
> ! Backend 2
> !
> -A KUBE-SEP-MMFZROQSLQ3DKOQA -s 57.112.0.248/32 -j KUBE-MARK-MASQ
> -A KUBE-SEP-MMFZROQSLQ3DKOQA -p tcp -m tcp -j DNAT --to-destination 57.112.0.248:8080
> !
> ! Service
> !
> -A KUBE-SERVICES -d 57.142.221.21/32 -p tcp -m comment --comment "default/app:http-web cluster IP" -m tcp --dport 80 -j KUBE-SVC-57XVOCFNTLTR3Q27
> !
> ! Load balancing between 2 backends
> !
> -A KUBE-SVC-57XVOCFNTLTR3Q27 -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-FS3FUULGZPVD4VYB
> -A KUBE-SVC-57XVOCFNTLTR3Q27 -j KUBE-SEP-MMFZROQSLQ3DKOQA
>
> I am looking for nftables equivalent for the load balancing part and also in this case there are double dnat translation, destination port from 80 to 8080 and destination IP: 57.112.0.247 or 57.112.0.248.
> Can it be expressed in a single nft dnat statement with vmaps or sets?
Regarding xt_statistic replacement, I once identified the equivalent of
'-m statistic --mode random --probability 0.5' would be 'numgen random
mod 0x2 < 0x1'.
Keeping both target address and port in a single map for *NAT statements
is not possible AFAIK.
If I'm not mistaken, you might be able to hook up a vmap together with
the numgen expression above like so:
| numgen random mod 0x2 vmap { \
| 0x0: jump KUBE-SEP-FS3FUULGZPVD4VYB, \
| 0x1: jump KUBE-SEP-MMFZROQSLQ3DKOQA }
Pure speculation, though. :)
Cheers, Phil
