Ok, I'm pretty sure there is something nasty going on inside nftables. I have repeated steps using nft and I got always the same errors like from libnftables. They are minor problems compared to large amounts of commands I pass through nft / libnftables but still I would appreciate to have them fixed. Password for zip archive is PassworD123. https://mega.nz/#!nnZDxaBa!P7kEBPhVvL-yOOUMlmA9XtdjVh4XFnGTGHHDg7BX_iA nftables-commands files are lists of commands which I use with nft -f and they are exact output of commands I use to generate and update the structure using libnftables. nftables-dump files with matching timestamp in name are state of nftables structure dumped using nft list ruleset before applying corresponding commands. Reproducible errors are: nftables-commands-2019-02-16 08:51:52.txt:20:2-33: Error: Could not process rule: Device or resource busy delete map ip filter group_6736 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:51:52.txt:21:2-38: Error: Could not process rule: Device or resource busy delete map ip filter group_6736_prio ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:51:52.txt:22:2-35: Error: Could not process rule: Device or resource busy delete chain ip filter group_6736 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:51:52.txt:29:2-34: Error: Could not process rule: No such file or directory flush chain ip filter group_6736 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ zsh: exit 1 nft -f nftables-commands-2019-02-16\ 08:51:52.txt nftables-commands-2019-02-16 08:52:08.txt:40:2-33: Error: Could not process rule: Device or resource busy delete map ip filter group_7737 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:08.txt:41:2-38: Error: Could not process rule: Device or resource busy delete map ip filter group_7737_prio ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:08.txt:42:2-35: Error: Could not process rule: Device or resource busy delete chain ip filter group_7737 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:08.txt:59:2-34: Error: Could not process rule: Device or resource busy delete map ip filter group_10271 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:08.txt:60:2-39: Error: Could not process rule: Device or resource busy delete map ip filter group_10271_prio ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:08.txt:61:2-36: Error: Could not process rule: Device or resource busy delete chain ip filter group_10271 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:08.txt:64:2-34: Error: Could not process rule: No such file or directory flush chain ip filter group_7737 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:08.txt:72:2-35: Error: Could not process rule: No such file or directory flush chain ip filter group_10271 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ zsh: exit 1 nft -f nftables-commands-2019-02-16\ 08:52:08.txt nftables-commands-2019-02-16 08:52:25.txt:4:2-62: Error: Could not process rule: No such file or directory delete element ip filter group_8164_prio { 10.143.10.64/27 } ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:25.txt:5:2-57: Error: Could not process rule: No such file or directory delete element ip filter group_8164 { 10.143.10.64/27 } ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:25.txt:10:2-33: Error: Could not process rule: Device or resource busy delete map ip filter group_8164 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:25.txt:11:2-38: Error: Could not process rule: Device or resource busy delete map ip filter group_8164_prio ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:25.txt:12:2-35: Error: Could not process rule: Device or resource busy delete chain ip filter group_8164 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nftables-commands-2019-02-16 08:52:25.txt:15:2-34: Error: Could not process rule: No such file or directory flush chain ip filter group_8164 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ zsh: exit 1 nft -f nftables-commands-2019-02-16\ 08:52:25.txt ---- S pozdravem / Best Regards Vaclav Zindulka On Fri, Feb 15, 2019 at 2:27 PM Václav Zindulka <vaclav.zindulka@xxxxxxxxxx> wrote: > > The behavior is random, as you can see. Here, delete element commands > were processed ok, but there were problems during flush chain so > referenced maps couldn't be deleted. > > 1 # fCreateRouter 10.5.59.166 > 2 add element ip filter router_map { 10.5.59.166 : "1:0x508" } > 3 # fCreateRouter 10.5.56.5 > 4 add element ip filter router_map { 10.5.56.5 : "1:0x509" } > 5 # fDeleteTarget 10.1.1.55 > 6 delete element ip filter group_3565_prio { 10.1.1.55/32 } > 7 delete element ip filter group_3565 { 10.1.1.55/32 } > 8 # fDeleteTarget 10.1.1.52 > 9 delete element ip filter group_3565_prio { 10.1.1.52/32 } > 10 delete element ip filter group_3565 { 10.1.1.52/32 } > 11 # fDeleteTarget 10.1.1.39 > 12 delete element ip filter group_3565_prio { 10.1.1.39/32 } > 13 delete element ip filter group_3565 { 10.1.1.39/32 } > 14 # fDeleteTarget 10.1.1.35 > 15 delete element ip filter group_3565_prio { 10.1.1.35/32 } > 16 delete element ip filter group_3565 { 10.1.1.35/32 } > 17 # fDeleteTarget 10.1.1.49 > 18 delete element ip filter group_3565_prio { 10.1.1.49/32 } > 19 delete element ip filter group_3565 { 10.1.1.49/32 } > 20 # fDeleteTarget 10.1.1.38 > 21 delete element ip filter group_3565_prio { 10.1.1.38/32 } > 22 delete element ip filter group_3565 { 10.1.1.38/32 } > 23 # fDeleteTarget 10.1.1.41 > 24 delete element ip filter group_3565_prio { 10.1.1.41/32 } > 25 delete element ip filter group_3565 { 10.1.1.41/32 } > 26 # fDeleteTarget 10.1.1.58 > 27 delete element ip filter group_3565_prio { 10.1.1.58/32 } > 28 delete element ip filter group_3565 { 10.1.1.58/32 } > 29 # fDeleteSubnet group_3565 10.1.1.32/27 > 30 delete element ip filter subnet_map {10.1.1.32/27} > 31 # fDeleteGroup group_3565 > 32 flush chain ip filter group_3565 > 33 delete map ip filter group_3565 > 34 delete map ip filter group_3565_prio > 35 delete chain ip filter group_3565 > 36 # fDeleteTarget 10.5.3.137 > 37 delete element ip filter group_8426_prio { 10.5.3.137/32 } > 38 delete element ip filter group_8426 { 10.5.3.137/32 } > 39 # fDeleteTarget 10.5.3.136 > 40 delete element ip filter group_8426_prio { 10.5.3.136/32 } > 41 delete element ip filter group_8426 { 10.5.3.136/32 } > 42 # fDeleteTarget 10.5.3.134 > 43 delete element ip filter group_8426_prio { 10.5.3.134/32 } > 44 delete element ip filter group_8426 { 10.5.3.134/32 } > 45 # fDeleteTarget 10.5.3.135 > 46 delete element ip filter group_8426_prio { 10.5.3.135/32 } > 47 delete element ip filter group_8426 { 10.5.3.135/32 } > 48 # fDeleteTarget 10.5.3.133 > 49 delete element ip filter group_8426_prio { 10.5.3.133/32 } > 50 delete element ip filter group_8426 { 10.5.3.133/32 } > 51 # fDeleteTarget 10.5.3.131 > 52 delete element ip filter group_8426_prio { 10.5.3.131/32 } > 53 delete element ip filter group_8426 { 10.5.3.131/32 } > 54 # fDeleteTarget 10.5.3.132 > 55 delete element ip filter group_8426_prio { 10.5.3.132/32 } > 56 delete element ip filter group_8426 { 10.5.3.132/32 } > 57 # fDeleteTarget 10.5.3.130 > 58 delete element ip filter group_8426_prio { 10.5.3.130/32 } > 59 delete element ip filter group_8426 { 10.5.3.130/32 } > 60 # fDeleteSubnet group_8426 10.5.3.128/28 > 61 delete element ip filter subnet_map {10.5.3.128/28} > 62 # fDeleteGroup group_8426 > 63 flush chain ip filter group_8426 > 64 delete map ip filter group_8426 > 65 delete map ip filter group_8426_prio > 66 delete chain ip filter group_8426 > 67 # fCreateGroup group_3565 > 68 # fReachableGroup group_3565 > 69 flush chain ip filter group_3565 > 70 add rule ip filter group_3565 meta priority 0 ip saddr > @priority_set meta priority set ip daddr map @group_3565_prio counter > 71 add rule ip filter group_3565 meta priority 0 ip daddr > @priority_set meta priority set ip saddr map @group_3565_prio counter > 72 add rule ip filter group_3565 meta priority 0 meta priority set > ip daddr map @group_3565 counter > 73 add rule ip filter group_3565 meta priority 0 meta priority set > ip saddr map @group_3565 counter > 74 add rule ip filter group_3565 meta priority 0 counter log prefix > "group_3565 - " > 75 # fCreateGroup group_8426 > 76 # fReachableGroup group_8426 > 77 flush chain ip filter group_8426 - I noticed my error here since > I'm trying to flush already deleted chain, but it demonstrates that > sometimes chains are deleted and sometimes not. > > /var/spool/shaperd/nftables.nft:33:1-32: Error: Could not process > rule: Device or resource busy > delete map ip filter group_3565 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > /var/spool/shaperd/nftables.nft:34:1-37: Error: Could not process > rule: Device or resource busy > delete map ip filter group_3565_prio > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > /var/spool/shaperd/nftables.nft:35:1-34: Error: Could not process > rule: Device or resource busy > delete chain ip filter group_3565 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > /var/spool/shaperd/nftables.nft:64:1-32: Error: Could not process > rule: Device or resource busy > delete map ip filter group_8426 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > /var/spool/shaperd/nftables.nft:65:1-37: Error: Could not process > rule: Device or resource busy > delete map ip filter group_8426_prio > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > /var/spool/shaperd/nftables.nft:66:1-34: Error: Could not process > rule: Device or resource busy > delete chain ip filter group_8426 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > /var/spool/shaperd/nftables.nft:69:1-33: Error: Could not process > rule: No such file or directory > flush chain ip filter group_3565 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > /var/spool/shaperd/nftables.nft:77:1-33: Error: Could not process > rule: No such file or directory > flush chain ip filter group_8426 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > ---- > S pozdravem / Best Regards > > Vaclav Zindulka > > On Fri, Feb 15, 2019 at 2:07 PM Václav Zindulka > <vaclav.zindulka@xxxxxxxxxx> wrote: > > > > > Works for me with nft from git: > > I've updated nftables to latest commit from git - I had it locked to > > last working state because of recent build problems. It still doesn't > > work for me from time to time. I'm attaching screenshot to show my > > problem with evidence item existed before deletion. I can provide > > whole structure including whole list of commands to create it if > > needed. Whole nftables structure contains 59250 rows. > > > > In the lower left part of the screenshot there is output of nft list > > ruleset before applying any changes. In the upper left part of screen > > there is list of commands to process and delete statements are first > > ones. They both fail. Also eighth command fails, which should process > > ok. I'm suspecting there could be some problem with flushing chain > > group_8165, because it references both maps so they can't be deleted > > but when I flush the chain by hand I can remove both maps. I've tried > > flushing other chain using libnftables and it works. In this case, > > however, it doesn't. Like it doesn't have enough time to process and > > then consequent commands fail too. In the right side area I'm checking > > existence of element in map by hand and trying to remove it. > > Unsuccessfully. Nft nor libnftables work in this case and many other > > similar cases. I'm using nft_run_cmd_from_filename function since it > > allows batch capabilities. > > > > /var/spool/shaperd/nftables.nft:2:1-61: Error: Could not process rule: > > No such file or directory > > delete element ip filter group_8165_prio { 10.143.10.96/27 } > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:3:1-56: Error: Could not process rule: > > No such file or directory > > delete element ip filter group_8165 { 10.143.10.96/27 } > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:8:1-32: Error: Could not process rule: > > Device or resource busy > > delete map ip filter group_8165 > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:9:1-37: Error: Could not process rule: > > Device or resource busy > > delete map ip filter group_8165_prio > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:10:1-34: Error: Could not process > > rule: Device or resource busy > > delete chain ip filter group_8165 > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:32:1-32: Error: Could not process > > rule: Device or resource busy > > delete map ip filter group_3564 > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:33:1-37: Error: Could not process > > rule: Device or resource busy > > delete map ip filter group_3564_prio > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:34:1-34: Error: Could not process > > rule: Device or resource busy > > delete chain ip filter group_3564 > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:36:1-62: Error: Could not process > > rule: No such file or directory > > delete element ip filter group_3569_prio { 192.168.124.5/32 } > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:37:1-57: Error: Could not process > > rule: No such file or directory > > delete element ip filter group_3569 { 192.168.124.5/32 } > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > /var/spool/shaperd/nftables.nft:42:1-32: Error: Could not process > > rule: Device or resource busy > > delete map ip filter group_3569 > > > > commands: > > 1 # fDeleteTarget 10.143.10.96 > > 2 delete element ip filter group_8165_prio { 10.143.10.96/27 } > > 3 delete element ip filter group_8165 { 10.143.10.96/27 } > > 4 # fDeleteSubnet group_8165 10.143.10.96/27 > > 5 delete element ip filter subnet_map {10.143.10.96/27} > > 6 # fDeleteGroup group_8165 > > 7 flush chain ip filter group_8165 > > 8 delete map ip filter group_8165 > > 9 delete map ip filter group_8165_prio > > 10 delete chain ip filter group_8165 > > > > nft list ruleset: > > map group_8165 { > > type ipv4_addr : classid > > flags interval > > elements = { 10.143.10.96/27 : 1:d656 } > > } > > > > map group_8165_prio { > > type ipv4_addr : classid > > flags interval > > elements = { 10.143.10.96/27 : 1:d657 } > > } > > > > > > there is prototype of structure I'm maintaining, but I have thousands > > of groups in two states. With 5 rules utilizing 2 maps and with one > > rule without maps. Sometimes I'm switching between states according to > > changes in routing tables. > > > > table ip filter { > > map subnet_map { > > type ipv4_addr : verdict > > flags interval > > elements = { 10.20.255.0/25 : goto group10, 10.20.255.128/25 : > > goto group11 } > > } > > > > map router_map { > > type ipv4_addr : classid > > elements = { 10.20.0.13 : 1:2 } > > } > > > > set priority_set { > > type ipv4_addr > > flags interval > > elements = { 10.20.2.1 } > > } > > > > map group10_prio { > > type ipv4_addr : classid > > flags interval > > elements = { 10.20.255.10 : 1:fffc, 10.20.255.14 : 1:fff9, > > 10.20.255.18 : 1:fff6 } > > } > > > > map group10 { > > type ipv4_addr : classid > > flags interval > > elements = { 10.20.255.10 : 1:fffb, 10.20.255.14 : 1:fff8, > > 10.20.255.18 : 1:ffe5 } > > } > > > > chain forward { > > type filter hook forward priority filter; policy accept; > > ip saddr 10.20.255.10 meta nftrace set 1 > > meta priority none ip daddr vmap @subnet_map counter packets 0 bytes 0 > > meta priority none ip saddr vmap @subnet_map counter packets 0 bytes 0 > > meta priority none meta priority set ip daddr map @router_map > > counter packets 0 bytes 0 > > meta priority none meta priority set ip saddr map @router_map > > counter packets 0 bytes 0 > > meta priority none counter packets 0 bytes 0 log prefix "forward - " > > } > > > > chain group11 { > > meta priority none meta priority set 1:b ip daddr 10.20.255.128/25 > > counter packets 0 bytes 0 > > meta priority none meta priority set 1:b ip saddr 10.20.255.128/25 > > counter packets 0 bytes 0 > > meta priority none counter packets 0 bytes 0 log prefix "group11 - " > > } > > > > chain group10 { > > meta priority none ip saddr @priority_set meta priority set ip > > daddr map @group10_prio counter packets 0 bytes 0 > > meta priority none ip daddr @priority_set meta priority set ip > > saddr map @group10_prio counter packets 0 bytes 0 > > meta priority none meta priority set ip daddr map @group10 counter > > packets 0 bytes 0 > > meta priority none meta priority set ip saddr map @group10 counter > > packets 0 bytes 0 > > meta priority none counter packets 0 bytes 0 log prefix "group10 - " > > } > > } > > ---- > > S pozdravem / Best Regards > > > > Vaclav Zindulka > > > > On Thu, Feb 14, 2019 at 12:18 PM Florian Westphal <fw@xxxxxxxxx> wrote: > > > > > > Václav Zindulka <vaclav.zindulka@xxxxxxxxxx> wrote: > > > > I've discovered problem with maps in nftables. When I try to remove > > > > last element of the map I get No such file or directory error. > > > > > > Works for me with nft from git: > > > > > > # nft list ruleset > > > table ip test { > > > map group_12058 { > > > type ipv4_addr : classid > > > flags interval > > > elements = { 10.13.25.32/29 : 1:b8a1 } > > > } > > > } > > > # nft delete element test group_12058 { 10.13.25.32/29 } > > > # nft list ruleset > > > table ip test { > > > map group_12058 { > > > type ipv4_addr : classid > > > flags interval > > > } > > > } > > > > > >
