On 2/25/25 1:58 AM, Robin Lee Powell wrote:
On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote:
On 2/21/25 7:02 PM, robinleepowell@xxxxxxxxx wrote:
So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES.
This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever.
Was the discussion on a public forum somewhere? I'd like to look at exactly
what they said.
Yep! Sorry, thought I linked to it, oops. https://lore.kernel.org/netfilter/132daf73-668f-4321-8945-c809db2277f5@xxxxxxxxxx/T/#t
Thanks! (I'm surprised I've never subscribed to that list, so that it
could be yet another folder with ever-increasing number of unread
messages :-/. Seriously though, I probably should be paying attention to it)
My one comment about their response/advice is that, while they suggest
that libvirt shouldn't be adding any firewall rules but should instead
just have docs telling people what rules they need to add, the entire
purpose of libvirt's virtual networks was to provide essentially a
single "That was Easy" button that can be pushed which sets up
*everything* needed for a guest to communicate with the outside - the
user can just say "create a network using NAT forwarding" and libvirt
creates the bridge device, sets up the DHCP and DNS servers listening on
that bridge device, turns on IP forwarding (if it's off) and also adds
all the rules necessary to allow the traffic to pass (and to NAT the
packets if that was requested). If we did "all those things *except* the
firewall rules" it would add another hurdle for inexperienced users to
get their VMs fully functional. (BTW, if someone wants that, they can
just define the network with <forward mode='open'/> and it will setup
everything except the firewall rules).
In my opinion, libvirt definitely *should* be adding the appropriate
rules (for the types of networks that are meant to be the most "plug and
play"), but also (and this is where the netfilter people are correct)
should do a better job of documenting what might need to be done beyond
that on some systems. In the meantime we should do our best to automate
(or at least specifically document) the "what might need to be done" for
as many other utilities as possible (for example the way we put all our
bridge devices in a special firewalld zone that we automatically add to
firewalld's configuration).
It's not *as nice* as the iptables situation, but having documentation that says "if you're using nftables make sure that packets with mark 79892 are accepted in all your chains" is quite straightforward compared to the current situation of "LOL good luck". (I'm not blaming anyone there!, the current situation is impossible for libvirt to navigate and it's not anyone's fault.)
It does still require that the other utilities know this secret number, and
agree to "anti-reject" it as we've requested, though. Also doesn't this
require that libvirt's table is processed first, before the other utilities'
tables? Otherwise, if the other tables are traversed before libvirt has a
chance to mark the packet with the special number, they won't get the
signal, so they'll reject the traffic. So I we would have set our table as a
higher priority, but then what if someone else sets their table with an
*even higher* priority? e.g. firewalld has "priority filter + 10" for its
forwarding rules, so could make ours "priority filter + 20", but what if,
e.g. docker decided to make theirs "priority filter +50"?). (yes, that's all
a rhetorical question. I guess in the end everything like this that we do
will chip away a bit more at the list of people who encounter problems; it
will never reach 0, but it will at least get closer :-))
Yep, those are all real concerns. :sigh:
Aside from that, libvirt's nftables rules are default accept, and it has no
rules looking at traffic that is destined for the host, only for forwarded
traffic that is going *through* the host, mainly with the intent of
rejecting stuff it doesn't like. So are you/they suggesting that this
forwarded traffic be marked with the special "libvirt code"? Or that we
should also add back rules that match input DNS/DHCP/TFTP on the
libvirt-created bridges, and have them both accept and mark those packets?
I think it'd have to be the latter to actually work.
If y'all don't like that, what's working excellently for me is adding `iifname "virbr*" accept` to my rule chain. FWIW.
Just keep in mind that "iifname" has to fetch the name of the interface and
do a string comparison for each packet, while "iif" just does a quick
comparison of ifindex, which I think is already saved away in the skb (of
course wildcards aren't possible in that case, but if you have just a couple
of libvirt networks it's still more efficient to have a rule using "iif" for
each interface.
The reason I have to use iifname is that at the time my rules are
loaded, the virbr interfaces *don't exist*. Like I actually have no
choice; it won't work the other way, unless I'm badly missing
something.
Yeah, good point. One alternate method you could experiment with if you
thought that the string compare was having anything more than a .0005%
effect on performance would be to use a libvirt network hook script
(https://libvirt.org/hooks.html ) - if you put your rule additions in
/etc/libvirt/hooks/network in the section run when argv[2] is "started"
then the bridge device (which you could grab out of the XML provided to
the script on stdin) would already be created. The down side is that
once you've added a hook script, all your networks are marked as
"tainted" which could create supportability problems if you were paying
someone for support; since you're using Fedora that's not a problem
though :-)
(I do recall someone at one point posting a perf chart where it showed a
significant amount of the kernel time on their system was spent doing
strcmp of the interface names. I don't remember exactly how much or
where it was posted, but I do remember being surprised at how much CPU
was used up for that...)
It was very hard to navigate through this situation because there's no documentation that this problem even exists.
Yeah, that's my fault. When I added the nftables backend, I forgot to update
https://libvirt.org/firewall.html (which is in docs/firewall.rst in the
libvirt sources). (also at the time I wrote the code, I I keep remembering
that I should do that, but only when I'm in the middle of something else and
somehow I haven't managed to even write it down on a list.
(I'm pretty sure there is a missing partial sentence in between the "I
I" up there, but no idea what that partial sentence was, which is an
example of the attention span that has led to (for example) this lack of
proper documentation :-P)
No attack intended; FOSS work is hard. :)
None perceived :-). Over the years there have been (fortunately not too
many, but certainly more than enough!) irate and hostile messages that
have gone by on this and other lists, but yours have not been in that
class - to the contrary, you're providing the necessary information for
us to understand your problem, doing your own investigation to help
narrow it down and better define it, and even volunteering to do some of
the work to fix it - that's the definition of being a good FOSS
community citizen :-) (oh, and I forgot the part about not
indiscriminately hurling insults around :-P)
My suggestion is to describe the situation at https://libvirt.org/firewall.html and suggest the virbr* fix, and down the road maybe look at this mark thing.
That's a kind of a broad solution though - libvirt's rules only reject
specific traffic between libvirt-created bridges (and incoming traffic from
outside a bridge's direct connects in the case of forward mode='nat'),
Anywhere they allow traffic, they allow *all* of it. The real problematic
stuff is traffic between the guests and the host (the rules we've had for
iptables that are absent in nftables are those to allow inbound DNS, DHCP,
and TFTP that are arriving on a virbr* interface, and destined for the
host). If you allow *all* traffic for virbr*, then you're leaving the host
wide open to all traffic from any guests (since libvirt's own rules are
default accept). I think the suggestion needs to be more than just "allow
all incoming on virbr*".
That's fair; I suppose we could post something equivalent to the old
iptables rules?
Yeah, and say something like "for minimal functionality and connection
to the outside"
And I do also like the idea of marking all the traffic that libvirt's
rules accept (and adding back the rules / marking the packets for the
basic functionality that our network setup provides). While it may not
work for everyone, it would likely be easier for someone to use the info
"accept all traffic with mark xyzzy" than to give out a list of all the
things that need to be accepted (and also more efficient, since there
would then be only a single extra rule in the other table no matter how
many libvirt networks were active).
I'll add this to my list of things to do and try to get to it this week
while it's fresh in my mind; I'll be sure to Cc you on the patches so
you'll know when there is something available to experiment with.
I'd like to help. I'm happy to write up issues for this, and I'm happy to write the updates to the firewall docs; just tell me what you'd like me to do.
firewall.rst should really be a shortened intro that links to the current
firewall.html for iptables (maybe renaming it "iptables.rst/html"?), and to
a new nftables.rst/html for information about nftables (including an
explanation of the "many tables, all must resolve to 'accept' problem.)
Since I've never gotten around to it in spite of wanting it done, I'd
certainly be happy to review an update done by anyone else :-)
Acknowledged. :)
