On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote: > On 2/21/25 7:02 PM, robinleepowell@xxxxxxxxx wrote: > > So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES. > > > > This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever. > > Was the discussion on a public forum somewhere? I'd like to look at exactly > what they said. > > > > > It's not *as nice* as the iptables situation, but having documentation that says "if you're using nftables make sure that packets with mark 79892 are accepted in all your chains" is quite straightforward compared to the current situation of "LOL good luck". (I'm not blaming anyone there!, the current situation is impossible for libvirt to navigate and it's not anyone's fault.) > > It does still require that the other utilities know this secret number, and > agree to "anti-reject" it as we've requested, though. Also doesn't this > require that libvirt's table is processed first, before the other utilities' > tables? Otherwise, if the other tables are traversed before libvirt has a > chance to mark the packet with the special number, they won't get the > signal, so they'll reject the traffic. So I we would have set our table as a > higher priority, but then what if someone else sets their table with an > *even higher* priority? e.g. firewalld has "priority filter + 10" for its > forwarding rules, so could make ours "priority filter + 20", but what if, > e.g. docker decided to make theirs "priority filter +50"?). (yes, that's all > a rhetorical question. I guess in the end everything like this that we do > will chip away a bit more at the list of people who encounter problems; it > will never reach 0, but it will at least get closer :-)) > > Aside from that, libvirt's nftables rules are default accept, and it has no > rules looking at traffic that is destined for the host, only for forwarded > traffic that is going *through* the host, mainly with the intent of > rejecting stuff it doesn't like. So are you/they suggesting that this > forwarded traffic be marked with the special "libvirt code"? Or that we > should also add back rules that match input DNS/DHCP/TFTP on the > libvirt-created bridges, and have them both accept and mark those packets? Yes, at the very least we need to add back the DNS/DHCP/TFTP rules with a packet mark, as those are rules that affect firewalling on the primary host NICs & thus most liable to be impacted by 3rd party firewalls. Possibly we could also mark all our other rules ACCEPT on virbr0 just in case someone has an especially strict firewall that would affect virbr0 too. Assuming packet marks don't have a performance hit, it wouldn't be hard for us to mark everything. > > My suggestion is to describe the situation at https://libvirt.org/firewall.html and suggest the virbr* fix, and down the road maybe look at this mark thing. > > That's a kind of a broad solution though - libvirt's rules only reject > specific traffic between libvirt-created bridges (and incoming traffic from > outside a bridge's direct connects in the case of forward mode='nat'), > Anywhere they allow traffic, they allow *all* of it. The real problematic > stuff is traffic between the guests and the host (the rules we've had for > iptables that are absent in nftables are those to allow inbound DNS, DHCP, > and TFTP that are arriving on a virbr* interface, and destined for the > host). If you allow *all* traffic for virbr*, then you're leaving the host > wide open to all traffic from any guests (since libvirt's own rules are > default accept). I think the suggestion needs to be more than just "allow > all incoming on virbr*". Yes, we only want to allow all incoming from virbr0 to the LAN, not to the host, otherwise you've bypassed the host protection With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
