On Wed, Jun 26, 2024 at 11:43:25PM GMT, Dorjoy Chowdhury wrote:
Hey Stefano,
Thanks a lot for all the details. I will look into them and reach out
if I need further input. Thanks! I have tried to summarize my
understanding below. Let me know if that sounds correct.
On Wed, Jun 26, 2024 at 2:37 PM Stefano Garzarella <sgarzare@xxxxxxxxxx> wrote:
Hi Dorjoy,
On Tue, Jun 25, 2024 at 11:44:30PM GMT, Dorjoy Chowdhury wrote:
>Hey Stefano,
[...]
>> >
>> >So the immediate plan would be to:
>> >
>> > 1) Build a new vhost-vsock-forward object model that connects to
>> >vhost as CID 3 and then forwards every packet from CID 1 to the
>> >Enclave-CID and every packet that arrives on to CID 3 to CID 2.
>>
>> This though requires writing completely from scratch the virtio-vsock
>> emulation in QEMU. If you have time that would be great, otherwise if
>> you want to do a PoC, my advice is to start with vhost-user-vsock which
>> is already there.
>>
>
>Can you give me some more details about how I can implement the
>daemon?
We already have a demon written in Rust, so I don't recommend you
rewrite one from scratch, just start with that. You can find the daemon
and instructions on how to use it with QEMU here [1].
>I would appreciate some pointers to code too.
I sent the pointer to it in my first reply [2].
>
>Right now, the "nitro-enclave" machine type (wip) in QEMU
>automatically spawns a VHOST_VSOCK device with the CID equal to the
>"guest-cid" machine option. I think this is equivalent to using the
>"-device vhost-vsock-device,guest-cid=N" option explicitly. Does that
>need any change? I guess instead of "vhost-vsock-device", the
>vhost-vsock device needs to be equivalent to "-device
>vhost-user-vsock-device,guest-cid=N"?
Nope, the vhost-user-vsock device requires just a `chardev` option.
The chardev points to the Unix socket used by QEMU to talk with the
daemon. The daemon has a parameter to set the CID. See [1] for the
examples.
>
>The applications inside the nitro-enclave VM will still connect and
>talk to CID 3. So on the daemon side, do we need to spawn a device
>that has CID 3 and then forward everything this device receives to CID
>1 (VMADDR_CID_LOCAL) same port and everything it receives from CID 1
>to the "guest-cid"?
Yep, I think this is right.
Note: to use VMADDR_CID_LOCAL, the host needs to load `vsock_loopback`
kernel module.
Before modifying the code, if you want to do some testing, perhaps you
can use socat (which supports both UNIX-* and VSOCK-*). The daemon for
now exposes two unix sockets, one is used to communicate with QEMU via
the vhost-user protocol, and the other is to be used by the application
to communicate with vsock sockets in the guest using the hybrid protocol
defined by firecracker. So you could initiate a socat between the latter
and VMADDR_CID_LOCAL, the only problem I see is that you have to send
the first string provided by the hybrid protocol (CONNECT 1234), but for
a PoC it should be ok.
I just tried the following and it works without touching any code:
shell1$ ./target/debug/vhost-device-vsock \
--vm guest-cid=3,socket=/tmp/vhost3.socket,uds-path=/tmp/vm3.vsock
shell2$ sudo modprobe vsock_loopback
shell2$ socat VSOCK-LISTEN:1234 UNIX-CONNECT:/tmp/vm3.vsock
shell3$ qemu-system-x86_64 -smp 2 -M q35,accel=kvm,memory-backend=mem \
-drive file=fedora40.qcow2,format=qcow2,if=virtio\
-chardev socket,id=char0,path=/tmp/vhost3.socket \
-device vhost-user-vsock-pci,chardev=char0 \
-object memory-backend-memfd,id=mem,size=512M \
-nographic
guest$ nc --vsock -l 1234
shell4$ nc --vsock 1 1234
CONNECT 1234
Note: the `CONNECT 1234` is required by the hybrid vsock protocol
defined by firecracker, so if we extend the vhost-device-vsock
daemon to forward packet to VMADDR_CID_LOCAL, that would not be
needed (including running socat).
Understood. Just trying to think out loud what the final UX will be
from the user perspective to successfully run a nitro VM before I try
to modify vhost-device-vsock to support forwarding to
VMADDR_CID_LOCAL.
I guess because the "vhost-user-vsock" device needs to be spawned
implicitly (without any explicit option) inside nitro-enclave in QEMU,
we now need to provide the "chardev" as a machine option, so the
nitro-enclave command would look something like below:
"./qemu-system-x86_64 -M nitro-enclave,chardev=char0 -kernel
/path/to/eif -chardev socket,id=char0,path=/tmp/vhost5.socket -m 4G
--enable-kvm -cpu host"
and then set the chardev id to the vhost-user-vsock device in the code
from the machine option.
Yep, it looks like an option. Maybe we can have
-M nitro-enclave,vhost-user-vsock=char0
The modified "vhost-device-vsock" would need to be run with the new
option that will forward everything to VMADDR_CID_LOCAL (below by the
"-z" I mean the new option)
"./target/debug/vhost-device-vsock -z --vm
IMHO the new option should be part of the --vm group (please avoid short
options) in conflict with `uds-path`. We may also need a parameter, e.g.
the CID where forward them.
Something like this:
--vm guest-cid=5,forward-cid=1,socket=/tmp/vhost5.socket
guest-cid=5,socket=/tmp/vhost5.socket,uds-path=/tmp/vm5.vsock"
this means the guest-cid of the nitro VM is CID 5, right?
Yep.
And the applications in the host would need to use VMADDR_CID_LOCAL
for communication instead of "guest-cid" (5) (assuming vsock_loopback
is modprobed). Let's say there are 2 applications inside the nitro VM
that connect to CID 3 on port 9000 and 9001. And the applications on
the host listen on 9000 and 9001 using VMADDR_CID_LOCAL. So, after the
commands above (qemu VM and vhost-device-vsock) are run, the
communication between the applications in the host and the
applications in the nitro VM on port 9000 and 9001 should just work,
right, without needing to run any extra socat commands or such?
Right.
or
will the user still need to run some socat commands for all the
relevant ports (e.g.,9000 and 9001)?
Nope, the "socat" work should be done by the vhost-device-vsock daemon.
I am just wondering what kind of changes are needed in
vhost-device-vsock for forwarding packets to VMADDR_CID_LOCAL? Will
that be something like this: the codepath that handles
"/tmp/vm5.vsock", upon receiving a "connect" (from inside the nitro
VM) for any port to "/tmp/vm5.vsock", vhost-device-vsock will just
connect to the same port using AF_VSOCK using the socket system calls
and messages received on that port in "/tmp/vm5.vsock" will be "send"
to the AF_VSOCK socket? or am I not thinking right and the
implementation would be something different entirely (change the CID
from 3 to 2 (or 1?) on the packets before they are handled then socat
will be needed probably)?
I think you're right.
Will this work if the applications in the
host want to connect to applications inside the nitro VM (as opposed
to applications inside the nitro VM connecting to CID 3)?
Nope, if you know in advance which ports to expose, you can add another
parameter to expose them, so the daemon con listen on the address
(cid=1, port=1234) and forwards the connections to the guest.
Stefano
