On Mon, Feb 24, 2025 at 10:58:37PM -0800, Robin Lee Powell wrote: > > > On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote: > > On 2/21/25 7:02 PM, robinleepowell@xxxxxxxxx wrote: > > > So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES. > > > > > > This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever. > > > > Was the discussion on a public forum somewhere? I'd like to look at exactly > > what they said. > > Yep! Sorry, thought I linked to it, oops. https://lore.kernel.org/netfilter/132daf73-668f-4321-8945-c809db2277f5@xxxxxxxxxx/T/#t > > > > It's not *as nice* as the iptables situation, but having documentation that says "if you're using nftables make sure that packets with mark 79892 are accepted in all your chains" is quite straightforward compared to the current situation of "LOL good luck". (I'm not blaming anyone there!, the current situation is impossible for libvirt to navigate and it's not anyone's fault.) > > > > It does still require that the other utilities know this secret number, and > > agree to "anti-reject" it as we've requested, though. Also doesn't this > > require that libvirt's table is processed first, before the other utilities' > > tables? Otherwise, if the other tables are traversed before libvirt has a > > chance to mark the packet with the special number, they won't get the > > signal, so they'll reject the traffic. So I we would have set our table as a > > higher priority, but then what if someone else sets their table with an > > *even higher* priority? e.g. firewalld has "priority filter + 10" for its > > forwarding rules, so could make ours "priority filter + 20", but what if, > > e.g. docker decided to make theirs "priority filter +50"?). (yes, that's all > > a rhetorical question. I guess in the end everything like this that we do > > will chip away a bit more at the list of people who encounter problems; it > > will never reach 0, but it will at least get closer :-)) > > Yep, those are all real concerns. :sigh: > > > Aside from that, libvirt's nftables rules are default accept, and it has no > > rules looking at traffic that is destined for the host, only for forwarded > > traffic that is going *through* the host, mainly with the intent of > > rejecting stuff it doesn't like. So are you/they suggesting that this > > forwarded traffic be marked with the special "libvirt code"? Or that we > > should also add back rules that match input DNS/DHCP/TFTP on the > > libvirt-created bridges, and have them both accept and mark those packets? > > I think it'd have to be the latter to actually work. > > > > If y'all don't like that, what's working excellently for me is adding `iifname "virbr*" accept` to my rule chain. FWIW. > > > > Just keep in mind that "iifname" has to fetch the name of the interface and > > do a string comparison for each packet, while "iif" just does a quick > > comparison of ifindex, which I think is already saved away in the skb (of > > course wildcards aren't possible in that case, but if you have just a couple > > of libvirt networks it's still more efficient to have a rule using "iif" for > > each interface. > > The reason I have to use iifname is that at the time my rules are > loaded, the virbr interfaces *don't exist*. Like I actually have no > choice; it won't work the other way, unless I'm badly missing > something. This could be a justification to mark *all* our allowed packets even on virbr0, not merely the host NIC, as you could then allow the marked packets without needing to know the NIC name or ID. Again, assuming packet marking is cheap from a performance POV ? With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
