* Tobin Feldman-Fitzthum (tobin@xxxxxxxxxxxxx) wrote: > > > On 3/3/22 12:20 PM, Daniel P. Berrangé wrote: > > On Fri, Feb 25, 2022 at 03:10:35PM -0500, Tobin Feldman-Fitzthum wrote: > >> > >> > >> On 2/24/22 7:26 AM, Daniel P. Berrangé wrote: > >>> On Wed, Feb 23, 2022 at 03:33:22PM -0500, Tobin Feldman-Fitzthum wrote: > >>>> > >>>> > >>>> On 2/23/22 1:38 PM, Dov Murik wrote: > >>>>> +cc Tobin, James > >>>>> > >>>>> On 23/02/2022 19:28, Daniel P. Berrangé wrote: > >>>>>> Extending management apps using libvirt to support measured launch of > >>>>>> QEMU guests with SEV/SEV-ES is unreasonably complicated today, both for > >>>>>> the guest owner and for the cloud management apps. We have APIs for > >>>>>> exposing info about the SEV host, the SEV guest, guest measurements > >>>>>> and secret injections. This is a "bags of bits" solution. We expect > >>>>>> apps to them turn this into a user facting solution. It is possible > >>>>>> but we're heading to a place where every cloud mgmt app essentially > >>>>>> needs to reinvent the same wheel and the guest owner will need to > >>>>>> learn custom APIs for dealing with SEV/SEV-ES for each cloud mgmt > >>>>>> app. This is pretty awful. We need to do a better job at providing > >>>>>> a solution that is more general purpose IMHO. > >>>>>> > >>> > >>>> Note in particular that we provide a client script called LaunchVM.py > >>>> that uses libvirt to start an SEV VM in conjunction with the attestation > >>>> server. This is basically a stand in for a management app or cloud > >>>> control plane. The modifications needed to launch an SEV VM are not > >>>> particularly extensive. I agree with some of your comments though. In > >>>> some ways it might be nice to have libvirt take care of things and hide > >>>> the complexity from the management app. > >>> > >>> LaunchVM.py nicely illustrates my concerns. Every application that > >>> uses libvirt that knows how to start VMs, now needs to be changed > >>> to support the series of operations shown in LaunchVM.py. THe guest > >>> owner probably can't use LaunchVM.py except for demoware, as they'll > >>> need a equivalent that talks to the API of their cloud mgmt app, > >>> of which there are many. > >>> > >>>> When we started working on our attestation server, our initial plan was > >>>> to make PRs to libvirt that would add one end of the attestation API to > >>>> libvirt, which would directly query the KBS. This is basically what you > >>>> are proposing. We decided against this for a couple of reasons. > >>>> > >>>> First, we were concerned that libvirt might not have network > >>>> connectivity to an arbitrary attestation server in a cloud environment. > >>>> We had envisioned that the guest owner would provide a URI for their > >>>> attestation server as part of the XML. This assumes that the node where > >>>> the VM is going to run can connect to an attestation server living > >>>> somewhere on the internet. I think that this might be challenging in > >>>> some cloud environments. By having the management app connect to libvirt > >>>> and the attestation server, we add some flexibility. > >>> > >>> Agreed, we can't assume that libvirt will always have ability to > >>> connect to an arbitrary service on the internet. > >>> > >>> That said, it does not neccessarily need this ability. If the user > >>> gives a URL of 'https://myhost.com/attest', the cloud doesn't have > >>> to give that straight to libvirt. The cloud software could have a > >>> attestation proxy server. So they could tell libvirt to use the > >>> URI https://10.0.0.5/attest, and then libvirt connects to that, > >>> it will proxy the calls through the guest owner's real attestation > >>> server. > >> > >> This might slightly contradict the idea of the management app being out > >> of the loop, but I guess setting up a proxy isn't very difficult. I > >> think CSPs already do this kind of thing to enable other features. > > > > The difference I see with a proxy approach is that it ought to end > > up being a dumb transport. It won't have to define any protocol or > > interpret the data, just blindly pass data back & forth. > > > > This is already something often done with VNC where the user connects > > to a public endpoint exposed by the cloud on its internet boundary, > > which then forwards the data onto QEMU's real VNC server on the > > compute host. In the VNC case, the public facing side often does > > websockets encapsulation, while the private side is pure VNC, but > > it still doesn't ened to understnd the VNC protocol so it is fairly > > easy to setup such a proxy. > > > > A proxy could also address the other problem I've realized. At least > > the first VM to be booted on a given cloud might be harder to attest > > because the attestation service would need to exist outside the cloud > > being used because the guest owner won't trust anything initially. > > > > This could imply that the attestation service is on a local machine > > controlled by the guest owner, even their local laptop. The implication > > is that the attestation service could easily be stuck behind NAT and > > be unable to accept incoming connections from the cloud. > > > > To address this the proxy might need to sit in the middle and accept > > incoming connections from both the guest owner's attestation service > > and from libvirt on the compute host. > > > > IOW, with VNC you get a unidirectional flow of connection > > establishment > > > > guest owner -> cloud proxy -> compute host > > > > but with the attestation service you might need a flow > > with the second arrow reversed ie > > > > guest owner -> cloud proxy <- compute host > > > > > Yeah I think a proxy is a reasonable solution. I am still not sure where > people are going to run their attestation servers in practice. We have > been assuming that they could be anywhere. I don't know if people will > insist on running them locally or if they will decide to trust some > managed attestation solution. I guess we will see. > > > Once the first VM is bootstrapped, it could be used to run an > > attestation service that is local to the cloud, avoiding the > > need for the traffic to traverse the internet when booting > > future VMs, and removing the dependancy on the guest owner having > > a local machine to run an attestation service on. > > > > > >>> The needs of all the apps look similar enough, because they are all > >>> ultimately constrained by the functionality made available by SEV(ES). > >>> Thus at least wrt apps doing traditional VM management using QEMU, > >>> it feels like it should be practical to come up with a common solution. > >>> I am all for common ground here. I had basically given up on it, but > >> maybe libvirt has enough influence to set a standard. In your first > >> email you said that there is a relatively short window to come up with > >> something and I think that is probably correct. > > > > Yes, there are a handful of major cloud software projects that use > > libvirt, and once they've all built their own solution, it won't be > > so easy to convince them to change into a shared solution, as it > > would create back compat problems for them. > > > >>> I can understand if it is harder to achieve commonality with tech > >>> like libkrun though, since that's consuming virt in quite a different > >>> way at the userspace level. > >> Yeah, extending the focus beyond SEV(-ES) with QEMU might make things > >> more difficult. There is some discussion right now about trying to find > >> common ground between SEV-SNP and TDX attestation, but I assume that is > >> all out of scope since libvirt isn't really involved. > > > > I admit I don't know much about TDX, but from what I've understood > > talking to other people, SEV-SNP might not end up looking all that > > different. IIUC the attestation has to be initiated from inside > > the SNP guest after CPUs are running. It is going need to be run > > as early as possible and while you might be able todo it in the > > initrd, it feels likely that it could be put into the firmware (OVMF) > > instead, such that it does the validation before even loading the > > kernel. This would facilitate supporting it with arbitrary guest OS, > > as the firmware is common to all. We can't assume the firmware will > > have direct network connectivity to any attestation service needed > > to verify the boot. This implies the firmware might need to talk > > to the host via something like virtio-serial / virtio-vsock, from > > where libvirt or QEMU can proxy the traffic onto the real attestation > > service. Such an architecture might end up aligning quite well with > > SEV/SEV-ES, possible allowing the same protocol to be used in both > > cases, just with differnt ultimate end points (libvirt for SEV(-ES) > > vs guest firmware for SEV-SNP). > > Yeah that is an interesting point. Most SNP approaches that I have seen > so far use the kernel/initrd to handle decryption. There is potentially > a gap if the kernel/initrd are not themselves part of the measurement > that is provided in the attestation report. We have been using this > measured direct boot thing for SEV(-ES) and I think it can be extended > to SEV-SNP as well. This would close that gap and make it feasible to do > the decryption in the kernel. With the direct boot setup, it feels like using 'clevis' in the initrd would be the right way to wire things to disk decryption. [ https://github.com/latchset/clevis ] It would need a 'pin' writing for SNP that then did whatever communication mechanism we settled on. (A clevis pin might also be the way to wire the simple disk key from your EFI/SEV mechanism up to LUKS? ) Dave > There might be reasons to do the measurement earlier, however. For > instance, it is easier to keep track of the hashes of fewer things (just > the firmware vs the firmware + initrd + kernel + cmdline). As you say > networking becomes a bit of an issue if you do the attestation in > firmware. Using a local device that is handled by libvirt could be a > good solution. > > I'm sure we could come up with a protocol that is general enough to > handle both pre-attestation and runtime attestation, but there > definitely are some differences. For instance with pre-attestation we > know that there will basically be two calls from management app to > attestation server. This is defined by the way that we inject secrets. > With SNP things are much more flexible. We can setup a persistent > connection between the guest and the attestation server and ask for > secrets throughout the runtime of the guest. It might take some thinking > to reconcile these approaches and could put some dependencies on how the > guest is supposed to behave, which isn't really our business (although a > firmware-based solution could be reasonable). > > -Tobin > > > > Regards, > > Daniel > -- Dr. David Alan Gilbert / dgilbert@xxxxxxxxxx / Manchester, UK
