Hey Stefano, On Wed, May 29, 2024 at 4:56 PM Stefano Garzarella <sgarzare@xxxxxxxxxx> wrote: > > On Wed, May 29, 2024 at 12:43:57PM GMT, Alexander Graf wrote: > > > >On 29.05.24 10:04, Stefano Garzarella wrote: > >> > >>On Tue, May 28, 2024 at 06:38:24PM GMT, Paolo Bonzini wrote: > >>>On Tue, May 28, 2024 at 5:53 PM Stefano Garzarella > >>><sgarzare@xxxxxxxxxx> wrote: > >>>> > >>>>On Tue, May 28, 2024 at 05:49:32PM GMT, Paolo Bonzini wrote: > >>>>>On Tue, May 28, 2024 at 5:41 PM Stefano Garzarella > >>>><sgarzare@xxxxxxxxxx> wrote: > >>>>>> >I think it's either that or implementing virtio-vsock in userspace > >>>>>> >(https://lore.kernel.org/qemu-devel/30baeb56-64d2-4ea3-8e53-6a5c50999979@xxxxxxxxxx/, > >>>>>> >search for "To connect host<->guest"). > >>>>>> > >>>>>> For in this case AF_VSOCK can't be used in the host, right? > >>>>>> So it's similar to vhost-user-vsock. > >>>>> > >>>>>Not sure if I understand but in this case QEMU knows which CIDs are > >>>>>forwarded to the host (either listen on vsock and connect to the host, > >>>>>or vice versa), so there is no kernel and no VMADDR_FLAG_TO_HOST > >>>>>involved. > >>>> > >>>>I meant that the application in the host that wants to connect to the > >>>>guest cannot use AF_VSOCK in the host, but must use the one where QEMU > >>>>is listening (e.g. AF_INET, AF_UNIX), right? > >>>> > >>>>I think one of Alex's requirements was that the application in the host > >>>>continue to use AF_VSOCK as in their environment. > >>> > >>>Can the host use VMADDR_CID_LOCAL for host-to-host communication? > >> > >>Yep! > >> > >>>If > >>>so, the proposed "-object vsock-forward" syntax can connect to it and > >>>it should work as long as the application on the host does not assume > >>>that it is on CID 3. > >> > >>Right, good point! > >>We can also support something similar in vhost-user-vsock, where instead > >>of using AF_UNIX and firecracker's hybrid vsock, we can redirect > >>everything to VMADDR_CID_LOCAL. > >> > >>Alex what do you think? That would simplify things a lot to do. > >>The only difference is that the application in the host has to talk to > >>VMADDR_CID_LOCAL (1). > > > > > >The application in the host would see an incoming connection from CID > >1 (which is probably fine) and would still be able to establish > >outgoing connections to the actual VM's CID as long as the Enclave > >doesn't check for the peer CID (I haven't seen anyone check yet). So > >yes, indeed, this should work. > > > >The only case where I can see it breaking is when you run multiple > >Enclave VMs in parallel. In that case, each would try to listen to CID > >3 and the second that does would fail. But it's a well solvable > >problem: We could (in addition to the simple in-QEMU case) build an > >external daemon that does the proxying and hence owns CID3. > > Well, we can modify vhost-user-vsock for that. It's already a daemon, > already supports different VMs per single daemon but as of now they have > to have different CIDs. > > > > >So the immediate plan would be to: > > > > 1) Build a new vhost-vsock-forward object model that connects to > >vhost as CID 3 and then forwards every packet from CID 1 to the > >Enclave-CID and every packet that arrives on to CID 3 to CID 2. > > This though requires writing completely from scratch the virtio-vsock > emulation in QEMU. If you have time that would be great, otherwise if > you want to do a PoC, my advice is to start with vhost-user-vsock which > is already there. > Can you give me some more details about how I can implement the daemon? I would appreciate some pointers to code too. Right now, the "nitro-enclave" machine type (wip) in QEMU automatically spawns a VHOST_VSOCK device with the CID equal to the "guest-cid" machine option. I think this is equivalent to using the "-device vhost-vsock-device,guest-cid=N" option explicitly. Does that need any change? I guess instead of "vhost-vsock-device", the vhost-vsock device needs to be equivalent to "-device vhost-user-vsock-device,guest-cid=N"? The applications inside the nitro-enclave VM will still connect and talk to CID 3. So on the daemon side, do we need to spawn a device that has CID 3 and then forward everything this device receives to CID 1 (VMADDR_CID_LOCAL) same port and everything it receives from CID 1 to the "guest-cid"? The applications that will be running in the host need to be changed so that instead of connecting to the "guest-cid" of the nitro-enclave VM, they will instead connect to VMADDR_CID_LOCAL. Is my understanding correct? BTW is there anything related to the "VMADDR_FLAG_TO_HOST" flag that needs to be checked? I remember some discussion about it. It would be great if you could give me some details about how I can achieve the CID 3 <-> CID 2 communication using the vhost-user-vsock. Is this https://github.com/stefano-garzarella/vhost-user-vsock where I would need to add support for forwarding everything to VMADDR_CID_LOCAL via an option maybe? Thanks and Regards, Dorjoy
