Greetings Laine, > Sent: Thursday, August 08, 2024 at 7:17 PM > From: "Laine Stump" <lstump@xxxxxxxxxx> > To: "users@xxxxxxxxxxxxxxxxx" <users@xxxxxxxxxxxxxxxxx> > Cc: "daggs" <daggs@xxxxxxx> > Subject: Re: cap seach for session libvirt > > On 8/8/24 2:40 AM, daggs wrote: > > Greetings Laine, > > > >> Sent: Thursday, August 08, 2024 at 6:16 AM > >> From: "Laine Stump" <lstump@xxxxxxxxxx> > >> To: users@xxxxxxxxxxxxxxxxx > >> Cc: "daggs" <daggs@xxxxxxx> > >> Subject: Re: cap seach for session libvirt > >> > >> On 8/7/24 2:25 PM, daggs via Users wrote: > >>> Greetings, > >>> > >>> I'm working on allowing a session vm to create a tap iface. > >>> the vm has this defintion: > >>> <interface type='ethernet'> > >> > >> If you were able to create a tap device with session-mode libvirt, what > >> would you do with it? It's not connected to anything, and your > >> session-mode libvirt wouldn't be able to set an IP address (which would > >> permit traffic to be *routed* to it by the host ip networking). > >> > >> > >>> <mac address='52:54:00:a7:79:6b'/> > >>> <target dev='veth0'/> > >> > >> Is there a particular reason you're wanting to name the device "veth0"? > >> That naming is usually use for veth devices (these are two-sided network > >> devices that are created in pairs, both looking like an ethernet. They > >> can be used to connect an LXC container running in a separate network > >> name space to the host's network name space (they are, as a matter of > >> fact, what libvirt's LXC driver uses to implement networking in LXC > >> containers). They are of really no use to a QEMU VM though. I'm pointing > >> this out because it's possible that the Linux kernel makes some > >> assumptions based on a device's name (unlikely, but also humans might > >> make incorrect assumptions based on the name). > >> > >> > >>> <model type='virtio'/> > >>> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> > >>> </interface> > >>> > >>> when I try to start the vm, I get this error: Unable to create tap device veth0: Operation not permitted > >>> searching the code led me to this line: https://github.com/libvirt/libvirt/blob/0caacf47d7b423db9126660fb0382ed56cd077c1/src/util/virnetdevtap.c#L232 > >>> I've looked on line and found out I need the net_admin_cap set. so I took the relevant code in to a dedicated test file and using pam_cap I've defined such cap > >> > >> The whole point of session mode libvirt is that it *can't* get all those > >> extra caps which would permit it (or the guest it runs) to wreak havoc > >> on the host by (for example) creating a tap device connected to some > >> random place. The way to get tap devices created for guests running > >> under session mode libvirt is to make sure the qemu-bridge-helper is > >> installed/setup properly on the host (including adding a line for the > >> bridge device you want to connect to into /etc/qemu/bridge.conf), then > >> just use <interface type='bridge'>, including a <source > >> bridge='yourbridgename'/>, but *not* including a <target dev='blah'/> > >> (since qemu-bridge-helper will determine the name for the tap device it > >> creates). > >> > >> > >> At guest start time. libvirt will see that a tap device is needed, then > >> exec qemu-bridge-helper (which is a suid-root binary with the necessary > >> caps to create a tap and connect it to a bridge) to create the tap > >> device and connect it to the specified bridge. It passes the name of the > >> tap device back to libvirt, which in turn provides the tap device to QEMU. > >> > >> > >> Of course your example suggests that you don't want the tap device > >> connected to a bridge, but in that case again I would ask "Why?". > >> > >>> for the test file, all went well. > >>> so I took it back to virsh and defined that cap to virsh but I'm still getting the same issue, see: https://ibb.co/zHggRQZ > >>> the os is debian 12 > >>> > >>> any ideas why I'm still getting this error? > >>> > >>> Thanks, > >>> > >>> Dagg > >>> > >> > > my goal is rather simple (at least for me), I have a server that runs two vms, one (denote vm1) is running openwrt, it has 5 nic via pt, > > > What is "pt"? Do you mean VFIO device assignment (i.e. "passthrough"?) yes, my apologies, it is a abbreviation I use in my workplace > > > > the other vm (denote vm2) runs libreelec. > > I need to allow both the host and vm2 access to the lan. > > > When you say "access the lan", do you mean that they need to be on the > same subnet as the physical lan? And are you implying here that the > openwrt VM should *not* be directly on the lan? no, as clarified above, the openwrt VM has 5 physical nics configured to be used with VFIO and passed into the vm. one of them is connected to the optical converted and openwrt is configured to use that device as wan. > > > > I think I was able to connect vm2 to vm1 using a virtual switch > > > What do you mean by "virtual switch"? A Linux host bridge? What is the > configuration of that bridge? Is it attached to a host physical > ethernet? what is the <interface> config of the two VMs? (AIn short - > "You haven't given me any details, and so I don't know what I'm working > with, and that makes it nearly impossible to tell you what to do next." :-)) my apologies, I'm bad with explanations. I've configured this network bridge in libvirt: <network> <name>default</name> <uuid>f90b3044-81c1-4c22-98df-8bbca3153f21</uuid> <bridge name='virsw0' stp='on' delay='0'/> <mac address='52:54:00:6b:1b:92'/> </network> this results with a bridge which doesn't provides any ip, like an physical unmanaged switch, hence virtual switch I'm connecting to it all the vms on the host, e.g. vm1 (openwrt VM) and vm2 (libreelec vm). the nic on vm2 communicates with vm1 and gets it ip from it (that is how my current setup is connected but everything run in the system level) > > > > > but I still need to connect the host to vm1. > > > If you are connecting vm2 to vm1 using a Linux host bridge, then all you > need to do is give that host bridge an IP address - then both vm1 and > vm2 can communicate with the host via that IP address. > I don't need both vm1 and vm2 to communicate with the host directly, I need only vm1 to do it. > > that is why I'm trying the tap way, I've tried to connect the host using the virtual switch but didn't got any ip from vm1. > > > What did you do to "try connecting the host using the virtual switch"? > Instead of a generalized summary, give me the exact changes you made to > libvirt and host system config. the bridge configuration can be seen above. vm1's xml has this: <interface type='bridge'> <mac address='52:54:00:53:1c:6b'/> <source bridge='virsw0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </interface> vm2's xml has this: <interface type='bridge'> <mac address='52:54:00:5a:4c:8c'/> <source bridge='virsw0'/> <model type='e1000e'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> this config works when both vms run in system namespace, now I'm trying to get it to work on session namespace. I've defined the bridge configuration above in system namespace and allowing it by adding "allow virsw0" to /etc/qemu/bridge.conf. > > > Also, this statement implies that you have a dhcp server running on vm1, > and that you want the host to get its IP from vm1's dhcp server. Is that > correct? If so, how were you planning to enable dhclient for the host > side of the tap device after creating it at runtime? This just doesn't > seem practically achievable. on the current setup that uses system namespace, I have this config: <interface type='ethernet'> <mac address='52:54:00:a7:79:6b'/> <target dev='veth0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> which results with the following device which gets ip: # ifconfig veth0 veth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 23.179.17.3 netmask 255.255.255.0 broadcast 23.179.17.255 inet6 fdab:9802:eb52:0:fc54:ff:fea7:796b prefixlen 64 scopeid 0x0<global> inet6 fe80::fc54:ff:fea7:796b prefixlen 64 scopeid 0x20<link> ether fe:54:00:a7:79:6b txqueuelen 1000 (Ethernet) RX packets 2185057 bytes 183502855 (175.0 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 499289 bytes 2528586323 (2.3 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > maybe my problem is that I'm not using qemu-bridge-helper, I tried to get it to work but wasn't able, > > > ... which implies that the connection you made between vm1 & vm2 was not > via a Linux host bridge, since qemu-bridge-helper is the only method of > connection a session mode libvirt guest to a Linux host bridge, so now > I'm just confused :-P. as said, I'm still working on the transition from system namespace to session. I didn't got to boot vm2 yet so I don't know if I even got any connection, my current goal is to connect the host to vm1 without the need for bridge that provides an ip. > > > What exactly was the <interface> config of the guest, and what what the > symptom(s) of "not working"? What was in the <interface> section of "virsh I'm unable to define an interface that connects the host to vm1 without the need for bridge that provides and ip in session namespace > > > > so I assume I either didn't configured the device properly in vm1 or didn't configured libvirt to use it properly. > > I used this tutorial: https://mike42.me/blog/2019-08-how-to-use-the-qemu-bridge-helper-on-debian-10. > > > I took a look there and that really is all there is to it - one > difference I see from the situation on Fedora is that the > qemu-bridge-helper is apparently in /usr/lib/qemu on debian, rather than > in /usr/libexec as it is on Fedora. Is that correct? thats correct, in debian it sits in /usr/lib/qemu/qemu-bridge-helper I didn't configured the variable in the config with absolute path, so that is why I'm saying that I might have misconfigured it. > > If you follow those instructions exactly, then your VMs should have IP > addresses on 192.168.112.0/24, as will the host (it will be visible to > the VMs at 192.168.122.1) - exactly the same as if you were using > qemu:///system and <interface type='network'><source network='default'/>. so I need to see the same ip in my scenario? > > > > I'll try again, maybe now it will work > > other ideas are welcomed > > > It would really help to see a drawing (either ASCII drawing directly in > email, or a link to an image of a drawing (either computer-generated or > hand-drawn) on imgur or some similar site (please don't send graphic > attachments to the mailing list :-)). This drawing should show 5 > entities (host, vm1, vm2, physical lan, internet connection) with lines > showing which interfaces of which machines should be on the same subnet. > Having that would greatly help in being able to give you productive advice. here: https://imgur.com/a/LilqljE I hope it is more understandable. n are physical nics, t are virtual nics. the switch is like an physical unmanaged switch but virtual. I have this configuration working in system namespace, I want to make it more isolated and migrate vm1 and vm2 to dedicated session namespace each
