On Thu, Aug 03, 2023 at 09:48:01 +0200, Stefano Garzarella wrote:
> On Wed, Aug 2, 2023 at 10:33 PM Jonathon Jongsma <jjongsma@xxxxxxxxxx> wrote:
> > On 7/24/23 8:05 AM, Peter Krempa wrote:
>
> [...]
>
> > >
> > > I've also noticed that using 'qcow2' format for the device doesn't work:
> > >
> > > error: internal error: process exited while connecting to monitor: 2023-07-24T12:54:15.818631Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage"}: Could not read qcow2 header: Invalid argument
> > >
> > > If that is supposed to work, then qemu devs will probably need to know
> > > about that, if that is not supposed to work, libvirt needs to add a
> > > check, because the error doesn't tell much. It's also possible I've
> > > messed up when formatting the image though, as didn't really try to
> > > figure out what's happening.
> > >
> >
> >
> > That's a good question, and I don't actually know the answer. Were you
> > using an actual vdpa block device for your tests or were you using the
> > vdpa block simulator kernel module? How did you set it up? Adding
> > Stefano to cc for his thoughts.
>
> Yep, I would also like to understand how you initialized the device
> with a qcow2 format.
Naively and originally I've simply used it as 'raw' at first and
formatted it from the guest OS. Then I've shut-down the VM and started
it back reconfiguring the image format as qcow2. This normally works
with real-file backed storage, and since the vdpa simulator seems to
persist the contents I supposed this would work.
> Theoretically, the best use case for vDPA block is that the backend
> handles formats, for QEMU it should just be a virtio device, but being
> a blockdev, we should be able to use formats anyway, so it should
> work.
Yeah, ideally there will be no format driver in qemu used for these
devices (this is not yet the case, I'll need to fix libvirt to stop
using the 'raw' driver if not needed).
Here I'm more interested whether it is supposed to work, in which case
we want to allow using qcow2 as a format in libvirt, or it's not
supposed to work and we should forbid it before the user gets a
suboptimal error message such as now.
>
> For now, waiting for real hardware, the only way to test vDPA block
> support in QEMU is to use the simulator in the kernel or VDUSE.
>
> With the kernel simulator we only have a 128 MB ramdisk available,
> with VDUSE you can use QSD with any file:
>
> $ modprobe -a vhost_vdpa vduse
> $ qemu-storage-daemon \
> --blockdev file,filename=/path/to/image.qcow2,cache.direct=on,aio=native,node-name=file
> \
> --blockdev qcow2,file=file,node-name=qcow2 \
> --export vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=qcow2,writable=on
>
> $ vdpa dev add name vduse0 mgmtdev vduse
>
> Then you have a /dev/vhost-vdpa-X device that you can use with the
> `virtio-blk-vhost-vdpa` blockdev (note: vduse requires QEMU with a
> memory-backed with `share=on`), but using raw since the qcow2 is
> handled by QSD.
> Of course, we should be able to use raw file with QSD and qcow2 on
> qemu (although it's not the optimal configuration), but I don't know
> how to initialize a `virtio-blk-vhost-vdpa` blockdev with a qcow2
> image :-(
With the above qemu storage daemon you should be able to do that by
simply dropping the qcow2 format driver and simply exposing a qcow2
formatted image. It similarly works with NBD:
I've formatted 2 qcow2 images:
# qemu-img create -f qcow2 /root/image1.qcow2 100M
# qemu-img create -f qcow2 /root/image2.qcow2 100M
And then exported them both via vduse and nbd without interpreting
qcow2, thus making the QSD into just a dumb storage device:
# qemu-storage-daemon \
--blockdev file,filename=/root/image1.qcow2,cache.direct=on,aio=native,node-name=file1 \
--export vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=file1,writable=on \
--blockdev file,filename=/root/image2.qcow2,cache.direct=on,aio=native,node-name=file2 \
--nbd-server addr.type=unix,addr.path=/tmp/nbd.sock \
--export nbd,id=nbd0,node-name=file2,writable=on,name=exportname
Now when I start a VM using the NBD export in qcow2 format:
<disk type='network' device='disk'>
<driver name='qemu' type='qcow2'/>
<source protocol='nbd' name='exportname'>
<host transport='unix' socket='/tmp/nbd.sock'/>
</source>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</disk>
The VM starts fine, but when using:
<disk type='vhostvdpa' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source dev='/dev/vhost-vdpa-0'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</disk>
I get:
error: internal error: QEMU unexpectedly closed the monitor (vm='vdpa'): 2023-08-07T12:34:21.628520Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage"}: Could not read qcow2 header: Invalid argument
What is weird though, that if I attempt to use qemu itself to format the
image, with VDPA simulator it appears to work the very first time after
the image is formatted, but if I restart the VM it no longer recognizes
the image. Since the simulator's content seems to be preserved accross
VM restarts when using 'raw' I thought it should work:
+ virsh start cd
Domain 'cd' started
+ virsh qemu-monitor-command cd --pass-fds 6 add-fd '{"fdset-id": 1337}'
{"return":{"fd":37,"fdset-id":1337},"id":"libvirt-441"}
+ virsh qemu-monitor-command cd blockdev-add '{"driver":"virtio-blk-vhost-vdpa","path":"/dev/fdset/1337","node-name":"testformat","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
{"return":{},"id":"libvirt-442"}
+ virsh qemu-monitor-command cd blockdev-create '{"options":{"driver":"qcow2","file":"testformat","size":10485760},"job-id":"formatjob"}'
{"return":{},"id":"libvirt-443"}
+ virsh qemu-monitor-command cd query-jobs
{"return":[{"current-progress":1,"status":"concluded","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-444"}
+ sleep 1
+ virsh qemu-monitor-command cd query-jobs
{"return":[{"current-progress":1,"status":"concluded","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-445"}
+ sleep 1
+ virsh qemu-monitor-command cd job-dismiss '{"id":"formatjob"}'
{"return":{},"id":"libvirt-446"}
+ virsh qemu-monitor-command cd blockdev-add '{"node-name":"testqcow2","read-only":false,"driver":"qcow2","file":"testformat"}'
{"return":{},"id":"libvirt-447"}
+ virsh destroy cd
Domain 'cd' destroyed
+ virsh qemu-monitor-command cd --pass-fds 6 add-fd '{"fdset-id": 1337}'
{"return":{"fd":37,"fdset-id":1337},"id":"libvirt-441"}
+ virsh qemu-monitor-command cd blockdev-add '{"driver":"virtio-blk-vhost-vdpa","path":"/dev/fdset/1337","node-name":"testformat","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
{"return":{},"id":"libvirt-442"}
+ virsh qemu-monitor-command cd blockdev-add '{"node-name":"testqcow2","read-only":false,"driver":"qcow2","file":"testformat"}'
{"id":"libvirt-443","error":{"class":"GenericError","desc":"Could not read qcow2 header: Invalid argument"}}
Additionally, even weirder is that if I use the qemu-storage-daemon
backed vhost-vdpa device, the formatting job simply gets stuck forever:
+ virsh start cd
Domain 'cd' started
+ virsh qemu-monitor-command cd blockdev-add '{"driver":"virtio-blk-vhost-vdpa","path":"/dev/vhost-vdpa-0","node-name":"testformat","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
{"return":{},"id":"libvirt-441"}
+ virsh qemu-monitor-command cd blockdev-create '{"options":{"driver":"qcow2","file":"testformat","size":10485760},"job-id":"formatjob"}'
{"return":{},"id":"libvirt-442"}
+ virsh qemu-monitor-command cd query-jobs
{"return":[{"current-progress":0,"status":"running","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-443"}
+ sleep 1
+ virsh qemu-monitor-command cd query-jobs
{"return":[{"current-progress":0,"status":"running","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-444"}
+ sleep 1
+ virsh qemu-monitor-command cd job-dismiss '{"id":"formatjob"}'
{"id":"libvirt-445","error":{"class":"GenericError","desc":"Job 'formatjob' in state 'running' cannot accept command verb 'dismiss'"}}
(Note, it behaves the same when using FD passing, I've just tried
because it's very weird)
Disclaimer: It's possible I broke my test box but I can't simply reboot
it at this point.
