HW Offload to Mellanox ConnectX-5

[Wojciech Wrona <w0jtas@xxxxxxxxxx>]

Hi, everyone. I'm having problems running nftables with hardware
offload. Every time keyword "flags offload" is added to flowtable
definition i get "Could not process rule: Operation not supported". For
now tried everything, i feel like nothing left to check, can anyone
point me the way ?

*My Config:*
OS: Ubuntu 20.04.6 LTS

Kernel: 5.15.0-73-generic #80~20.04.1-Ubuntu SMP Wed May 17 14:58:14 UTC
2023 x86_64 x86_64 x86_64 GNU/Linux

Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
(MCX516A-CCAT)

Device driver: mlx5_core
In fact i tried driver included in kernel and also compiled from sources
latest 5.8-2.0.3.0 version from Nvidia/Mellanox - same result on both.

root@nat-router2004:~# mlxburn -query -d /dev/mst/mt4119_pciconf0
-I- Image type:            FS4
-I- FW Version:            16.35.2000
-I- FW Release Date:       24.11.2022
-I- Product Version:       16.35.2000
-I- Rom Info:              type=UEFI version=14.28.16 cpu=AMD64
-I-                        type=PXE version=3.6.805 cpu=AMD64
-I- Description:           UID                GuidsNumber
-I- Base GUID:             1070fd0300b346d6        8
-I- Base MAC:              1070fdb346d6            8
-I- Image VSD:             N/A
-I- Device VSD:            N/A
-I- PSID:                  MT_0000000012
-I- Security Attributes:   N/A

root@nat-router2004:~# ethtool -k eth1 | grep tc-off
hw-tc-offload: on
root@nat-router2004:~# ethtool -k eth2 | grep tc-off
hw-tc-offload: on

root@nat-router2004:~# nft --version
nftables v1.0.7 (Old Doc Yak)
*This one was compiled from sources.*

root@nat-router2004:~# conntrack --version
conntrack v1.4.7 (conntrack-tools)
*This one was compiled from sources.*

nftables.conf
#!/usr/local/sbin/nft -f

flush ruleset

table ip filter {
    flowtable f {
    hook ingress priority filter; devices = { eth1, eth2 };
    flags offload;
    }

    chain input {
    type filter hook input priority filter; policy drop;
    ct state { related, established } accept
    iif "lo" accept
    iif "eth0" accept
    ip protocol icmp accept
    }

    chain forward {
    type filter hook forward priority filter; policy accept;
    ip protocol { tcp, udp } flow add @f
    }

    chain output {
    type filter hook output priority filter; policy accept;
    }
}

When i remove "flags offload" definition from flowtable everything works
fine, but in "conntrack -L" i see only [OFFLOAD] sign, instead of
[HW_OFFLOAD].

Anyone managed to force it to work as intended ? I am seeing only same
kind of questions on different internet maling lists but every subject
ends without any solution.

Best regards,

-- 
Wojciech Wrona