On 6/17/24 02:02, Daniel P. Berrangé wrote:
On Thu, Jun 13, 2024 at 08:06:17PM -0700, melanie witt wrote:
Hi,
I have been trying to use the librbd engine to run a guest from an encrypted
RBD image and am running into some problems.
What I would like to do is:
1. Start from an unencrypted raw image with an OS
2. Make an encrypted clone of that image
3. Boot a guest from the encrypted clone image
What I have tried so far (simplified):
1. Make a clone of the unencrypted image
rbd clone images/unencrypted@snap images/encryptedclone
2. Format the clone image with encryption
rbd encryption format images/encryptedclone luks1 passphrase.bin
3. Create guest XML with the encrypted clone
[...]
<disk type="network" device="disk">
<driver type="raw" cache="writeback"/>
<source protocol="rbd" name="images/encryptedclone">
<host name="127.0.0.1" port="6789"/>
<encryption format="luks" engine="librbd">
<secret type="passphrase" uuid="secretuuid"/>
</encryption>
</source>
<auth username="cinder">
<secret type="ceph" uuid="othersecretuuid"/>
</auth>
<target dev="vda" bus="virtio"/>
</disk>
[...]
and virDomainCreateWithFlags() with the XML.
I don't get any errors from libvirt (no errors about loading encryption) but
this configuration does not seem to work, the guest won't boot.
If anyone can give me a hint what I'm doing wrong, I would appreciate it.
Can you share the corresponding QEMU command line that gets generated.
Hi, thank you for replying! I did some more debugging over the weekend
and finally found the problem was that I had needed to resize the
unencrypted raw image first before cloning it to account for the LUKS
header, otherwise the image content ends up truncated after formatting
and the guest unbootable.
This concept is actually shown in an example in the docs
https://docs.ceph.com/en/reef/rbd/rbd-encryption/#examples but I still
had not made the connection between the guest not booting and the image
size.
So the working steps I currently have are:
1. Start with an unencrypted raw image with an OS
2. Resize the unencrypted raw image larger (example: current size + 1G)
3. Make a clone of the unencrypted image (snapshot + protect + clone)
4. Resize the unencrypted raw image back down to its original size
(current_size)
5. Format the clone image with encryption
6. Resize the clone image down to the original image size (current_size)
7. Create guest XML with the encrypted clone
The generated QEMU command line (which I assume has nothing wrong in it)
for the unbootable guest was:
2024-06-15 01:32:00.122+0000: starting up libvirt version: 9.6.0,
package: 9.6.0-1ubuntu1.1 (Ubuntu), qemu version: 8.0.4Debian
1:8.0.4+dfsg-1ubuntu3.23.10.5, kernel: 6.5.0-35-generic, hostname:
controller
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
\
HOME=/var/lib/libvirt/qemu/domain-4-instance-00000005 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-4-instance-00000005/.local/share
\
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-4-instance-00000005/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-4-instance-00000005/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=instance-00000005,debug-threads=on \
-S \
-object
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-4-instance-00000005/master-key.aes"}'
\
-machine
pc-i440fx-8.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=on \
-accel tcg \
-cpu Nehalem \
-m size=524288k \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":536870912}' \
-overcommit mem-lock=off \
-smp 1,sockets=1,dies=1,cores=1,threads=1 \
-uuid 3f863740-3d6e-4469-aa34-035cd7384f9e \
-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=29.1.0,serial=3f863740-3d6e-4469-aa34-035cd7384f9e,uuid=3f863740-3d6e-4469-aa34-035cd7384f9e,family=Virtual
Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=29,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-object
'{"qom-type":"secret","id":"libvirt-1-storage-auth-secret0","data":"VaS1qYDjnD5GACQOf+PRhdeH9MPizc4/K1jPJ3LCo0c=","keyid":"masterKey0","iv":"Hv/U+wms23aTK0GLnwn9Aw==","format":"base64"}'
\
-object
'{"qom-type":"secret","id":"libvirt-1-format-encryption-secret0","data":"KQn7lDD7TaPBdBHBPhIR8ytmRcPzVXjjdXbxlHhurSZceM8iYUyBgZwhoLADBD9M2v3vPvBiyFHH4PFhBjjjXC6cd8Zj8m3gI9gPkoqvkmCFh/LNGuDmeh2hIw6Ts0iO","keyid":"masterKey0","iv":"0ByebFRNgtGZcN7JarPKnQ==","format":"base64"}'
\
-blockdev
'{"driver":"rbd","pool":"vms","image":"3f863740-3d6e-4469-aa34-035cd7384f9e_disk","server":[{"host":"127.0.0.1","port":"6789"}],"encrypt":{"format":"luks","key-secret":"libvirt-1-format-encryption-secret0"},"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-auth-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}'
\
-device
'{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1,"write-cache":"on"}'
\
-netdev '{"type":"tap","fd":"32","id":"hostnet0"}' \
-device
'{"driver":"virtio-net-pci","host_mtu":1442,"netdev":"hostnet0","id":"net0","mac":"fa:16:3e:48:7b:21","bus":"pci.0","addr":"0x3"}'
\
-add-fd set=0,fd=30,opaque=serial0-log \
-chardev pty,id=charserial0,logfile=/dev/fdset/0,logappend=on \
-device
'{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,audiodev=audio1 \
-device
'{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pci.0","addr":"0x2"}'
\
-device
'{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x5"}'
\
-object
'{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device
'{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.0","addr":"0x6"}'
\
-device '{"driver":"vmcoreinfo"}' \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/4 (label charserial0)
