Network disks and replacing qemu-block-curl|ssh with nbdkit

[Jonathon Jongsma <jjongsma@xxxxxxxxxx>]

As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2016527, 
RHEL is planning to remove dependencies on the qemu-block-curl and 
qemu-block-ssh plugins from the main qemu package. This creates issues 
for libvirt for supporting network disk sources. So I've been looking 
into using nbdkit from libvirt to proxy these network disks to qemu as 
NBD disks.

The basic idea is that libvirt will spin up an nbdkit instance for e.g. 
an https network disk source, and will provide the resulting unix socket 
to qemu as an nbd disk. This allows libvirt to continue supporting 
http/ftp/ssh disk sources regardless of whether the qemu block plugins 
are installed or not.

However, there are a couple of issues and feature gaps that I've run 
into that I'd like to discuss.

1. secrets

There is some code in libvirt[1] which seems to expect that it is 
possible for http(s) disk sources to have a username and password 
specified. However, I can't find any valid xml schema for specifying an 
http username and password, and my reading of the code suggests that 
there shouldn't be any way for these to be set for http(s)/ftp(s) disk 
sources either since auth is only supported for ISCSI and RBD protocols 
[2]. Am I missing something?

[1] 
https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a836fd2e/src/qemu/qemu_block.c#L749

[2] 
https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a836fd2e/src/qemu/qemu_domain.c#L1235

If it *is* possible for the username/password to be set for these disks, 
then we have the issue that these sensitive pieces of data have so far 
been passed as encrypted data to qemu using qemu secrets. But if nbdkit 
is handling the http requests, we need to pass this data to nbdkit 
rather than qemu and so we can no longer use qemu secrets. The same 
issue applies to http cookies, which could potentially include 
security-sensitive data such as login credentials, etc.

Fortunately, nbdkit provides a method for reading cookies and passwords 
from a file, which should be secure if the file has permissions set 
properly. So I'm currently planning to write a file containing the 
cookies and pass them to nbdkit by specifying the filename. But I'm 
still confused about the username/password possibility.

2. readahead

The libvirt xml format allows to specify a readahead size for disks that 
are handled by the qemu-block-curl plugin. Unfortunately, nbdkit doesn't 
currently support any readahead configuration. In nbdkit, readahead is 
handled by an nbkit "filter" that takes no configuration options 
(`nbdkit --filter=readahead ...`). In theory, this filter tries to 
adaptively read ahead. But when I discussed it with Rich, he suggested 
that he had stopped using it in virt-v2v because it was causing more 
trouble than it was worth. He also suggested that this readahead filter 
might need a complete rewrite, and presumably the rewrite could include 
the ability to configure a readahead buffer size. But I'm not sure what 
the timeframe might be for that.

3. blockdev-create

There is support in libvirt[3] for creating ssh network disks by sending 
a 'blockdev-create' command to qemu. If qemu is no longer handling ssh 
network disk sources directly, this feature becomes significantly more 
complicated. I don't yet know enough about this part of the libvirt code 
to know what further complications might pop up here. From my reading of 
the code, this is mostly triggered by things like `virsh blockcopy` 
`virsh backup-begin`, etc. But nbdkit cannot currently do this. Rich 
pointed me to a recent commit[4] where he added disk creation to the 
nbdkit vddk plugin, and suggested that something similar could be added 
for the nbdkit ssh plugin.

[3] 
https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a836fd2e/src/qemu/qemu_block.c#L2639

[4] 
https://gitlab.com/nbdkit/nbdkit/-/commit/a39d5773afc3ebab7e5768118a2bccb89a654585


It seems to me that it's essential that we resolve #3 before we can move 
forward with nbdkit support in libvirt. (Although I admit that I have no 
idea how common it is for people to use ssh disks so I suppose there's a 
slim possibility that we could just disable the 'create disk' feature 
for ssh disks without any practical loss of functionality?) But it's 
less obvious to me whether we could move ahead despite missing readahead 
size configuration, etc.

Thoughts?

Jonathon