On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote: > On 2/21/25 7:02 PM, robinleepowell@xxxxxxxxx wrote: > > So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES. > > > > This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever. > > Was the discussion on a public forum somewhere? I'd like to look at exactly > what they said. Yep! Sorry, thought I linked to it, oops. https://lore.kernel.org/netfilter/132daf73-668f-4321-8945-c809db2277f5@xxxxxxxxxx/T/#t > > It's not *as nice* as the iptables situation, but having documentation that says "if you're using nftables make sure that packets with mark 79892 are accepted in all your chains" is quite straightforward compared to the current situation of "LOL good luck". (I'm not blaming anyone there!, the current situation is impossible for libvirt to navigate and it's not anyone's fault.) > > It does still require that the other utilities know this secret number, and > agree to "anti-reject" it as we've requested, though. Also doesn't this > require that libvirt's table is processed first, before the other utilities' > tables? Otherwise, if the other tables are traversed before libvirt has a > chance to mark the packet with the special number, they won't get the > signal, so they'll reject the traffic. So I we would have set our table as a > higher priority, but then what if someone else sets their table with an > *even higher* priority? e.g. firewalld has "priority filter + 10" for its > forwarding rules, so could make ours "priority filter + 20", but what if, > e.g. docker decided to make theirs "priority filter +50"?). (yes, that's all > a rhetorical question. I guess in the end everything like this that we do > will chip away a bit more at the list of people who encounter problems; it > will never reach 0, but it will at least get closer :-)) Yep, those are all real concerns. :sigh: > Aside from that, libvirt's nftables rules are default accept, and it has no > rules looking at traffic that is destined for the host, only for forwarded > traffic that is going *through* the host, mainly with the intent of > rejecting stuff it doesn't like. So are you/they suggesting that this > forwarded traffic be marked with the special "libvirt code"? Or that we > should also add back rules that match input DNS/DHCP/TFTP on the > libvirt-created bridges, and have them both accept and mark those packets? I think it'd have to be the latter to actually work. > > If y'all don't like that, what's working excellently for me is adding `iifname "virbr*" accept` to my rule chain. FWIW. > > Just keep in mind that "iifname" has to fetch the name of the interface and > do a string comparison for each packet, while "iif" just does a quick > comparison of ifindex, which I think is already saved away in the skb (of > course wildcards aren't possible in that case, but if you have just a couple > of libvirt networks it's still more efficient to have a rule using "iif" for > each interface. The reason I have to use iifname is that at the time my rules are loaded, the virbr interfaces *don't exist*. Like I actually have no choice; it won't work the other way, unless I'm badly missing something. > > It was very hard to navigate through this situation because there's no documentation that this problem even exists. > > Yeah, that's my fault. When I added the nftables backend, I forgot to update > https://libvirt.org/firewall.html (which is in docs/firewall.rst in the > libvirt sources). (also at the time I wrote the code, I I keep remembering > that I should do that, but only when I'm in the middle of something else and > somehow I haven't managed to even write it down on a list. No attack intended; FOSS work is hard. :) > > My suggestion is to describe the situation at https://libvirt.org/firewall.html and suggest the virbr* fix, and down the road maybe look at this mark thing. > > That's a kind of a broad solution though - libvirt's rules only reject > specific traffic between libvirt-created bridges (and incoming traffic from > outside a bridge's direct connects in the case of forward mode='nat'), > Anywhere they allow traffic, they allow *all* of it. The real problematic > stuff is traffic between the guests and the host (the rules we've had for > iptables that are absent in nftables are those to allow inbound DNS, DHCP, > and TFTP that are arriving on a virbr* interface, and destined for the > host). If you allow *all* traffic for virbr*, then you're leaving the host > wide open to all traffic from any guests (since libvirt's own rules are > default accept). I think the suggestion needs to be more than just "allow > all incoming on virbr*". That's fair; I suppose we could post something equivalent to the old iptables rules? > > I'd like to help. I'm happy to write up issues for this, and I'm happy to write the updates to the firewall docs; just tell me what you'd like me to do. > > firewall.rst should really be a shortened intro that links to the current > firewall.html for iptables (maybe renaming it "iptables.rst/html"?), and to > a new nftables.rst/html for information about nftables (including an > explanation of the "many tables, all must resolve to 'accept' problem.) > > Since I've never gotten around to it in spite of wanting it done, I'd > certainly be happy to review an update done by anyone else :-) Acknowledged. :)
