Hi, users@,
I'm attempting to set up domain with SGX EPC memory. When starting it, I get
this error:
# virsh start astoria-test-1
error: Failed to start domain 'astoria-test-1'
error: internal error: process exited while connecting to monitor: libnuma: Warning: /sys not mounted or invalid. Assuming one node: Permission denied
2023-11-16T14:04:26.919285Z qemu-system-x86_64: The nodes select no CPUs
When removing <memory model='sgx-epc'> node from XML, the domain starts fine.
I've attached is domain.xml for it, with the section commented. Also there are
logs from happy and sad case.
The libnuma error seems to stem from this place:
https://github.com/numactl/numactl/blob/fc999282e6663d6844773e6065bbee3270f36c19/libnuma.c#L1375
This makes no sense to me, because the file is there and available:
# sudo -u libvirt-qemu cat /sys/devices/system/node/node0/cpumap
fff
I also don't understand why qemu says "The nodes select no CPUs"
(https://gitlab.com/qemu-project/qemu/-/blob/34a5cb6d8434303c170230644b2a7c1d5781d197/util/thread-context.c#L212).
DDG has failed me in all cases.
I've also tried to follow the recommendation from
https://libvirt.org/formatdomaincaps.html#sgx-capabilities:
For more details on the SGX feature, please follow resources in the SGX
developer's document store.
...and the only think that I've found relevant to libvirt was this document:
https://www.intel.com/content/www/us/en/developer/articles/technical/virtualizing-intel-software-guard-extensions-with-kvm-and-qemu.html
which suggests to play with <qemu:commandline> instead.
host: Debian 12, libvirt 9.0.0, QEMU 7.2, virtinst 4.1 (see dpkg-l attachment)
guest: Ubuntu 22.04 (jammy) cloud image
The domain in quesion was defined using virt-install with cloud-init, then
I was editing XML with `virsh edit`.
Relevant snippet from `virsh domcapabilities`:
<sgx supported='yes'>
<flc>yes</flc>
<sgx1>yes</sgx1>
<sgx2>no</sgx2>
<section_size unit='KiB'>95744</section_size>
<sections>
<section node='0' size='95744' unit='KiB'/>
</sections>
</sgx>
System-wide logs do not contain anything relevant. I'm happy to provide
additional info if needed.
Anyone can point me to a solution?
--
pozdrawiam / best regards
Wojtek Porczyk
Gramine / Invisible Things Lab
I do not fear computers,
I fear lack of them.
-- Isaac Asimov
Attachment:
astoria-test-1.xml
Description: XML document
# dpkg -l \*virt\* \*qemu\* \*numa\* | fgrep ii ii ipxe-qemu 1.0.0+git-20190125.36a4c85-5.1 all PXE boot firmware - ROM images for qemu ii libnuma1:amd64 2.0.16-1 amd64 Libraries for controlling NUMA policy ii libvirt-clients 9.0.0-4 amd64 Programs for the libvirt library ii libvirt-daemon 9.0.0-4 amd64 Virtualization daemon ii libvirt-daemon-config-network 9.0.0-4 all Libvirt daemon configuration files (default network) ii libvirt-daemon-config-nwfilter 9.0.0-4 all Libvirt daemon configuration files (default network filters) ii libvirt-daemon-driver-lxc 9.0.0-4 amd64 Virtualization daemon LXC connection driver ii libvirt-daemon-driver-qemu 9.0.0-4 amd64 Virtualization daemon QEMU connection driver ii libvirt-daemon-driver-vbox 9.0.0-4 amd64 Virtualization daemon VirtualBox connection driver ii libvirt-daemon-driver-xen 9.0.0-4 amd64 Virtualization daemon Xen connection driver ii libvirt-daemon-system 9.0.0-4 amd64 Libvirt daemon configuration files ii libvirt-daemon-system-systemd 9.0.0-4 all Libvirt daemon configuration files (systemd) ii libvirt-l10n 9.0.0-4 all localization for the libvirt library ii libvirt0:amd64 9.0.0-4 amd64 library for interfacing with different virtualization systems ii numactl 2.0.16-1 amd64 NUMA scheduling and memory placement tool ii numad 0.5+20150602-8+b1 amd64 User-level daemon that monitors NUMA topology and usage ii python3-libvirt 9.0.0-1 amd64 libvirt Python 3 bindings ii qemu-block-extra 1:7.2+dfsg-7+deb12u2 amd64 extra block backend modules for qemu-system and qemu-utils ii qemu-system-common 1:7.2+dfsg-7+deb12u2 amd64 QEMU full system emulation binaries (common files) ii qemu-system-data 1:7.2+dfsg-7+deb12u2 all QEMU full system emulation (data files) ii qemu-system-gui 1:7.2+dfsg-7+deb12u2 amd64 QEMU full system emulation binaries (user interface and audio support) ii qemu-system-x86 1:7.2+dfsg-7+deb12u2 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 1:7.2+dfsg-7+deb12u2 amd64 QEMU utilities ii virtinst 1:4.1.0-2 all utilities to create and edit virtual machines
2023-11-16 13:43:54.764+0000: starting up libvirt version: 9.0.0, package: 9.0.0-4 (Debian), qemu version: 7.2.5Debian 1:7.2+dfsg-7+deb12u2, kernel: 6.1.0-13-amd64, hostname: astoria.[snip]
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=astoria-test-1,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-astoria-test-1/master-key.aes"}' \
-machine pc-q35-7.2,usb=off,dump-guest-core=off,sgx-epc.0.memdev=memepc0,sgx-epc.0.node=0 \
-accel kvm \
-cpu host,migratable=on \
-m size=4194304k,slots=1,maxmem=4210688k \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-object '{"qom-type":"memory-backend-ram","id":"ram-node0","size":4294967296,"host-nodes":[0],"policy":"bind"}' \
-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-uuid e74129a0-2e80-45af-88d4-e6949d7dd628 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"pcie-root-port","port":15,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x7"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"}' \
-object '{"qom-type":"thread-context","id":"tc-memepc0","node-affinity":[0]}' \
-object '{"qom-type":"memory-backend-epc","id":"memepc0","prealloc":true,"size":16777216,"host-nodes":[0],"policy":"bind","prealloc-context":"tc-memepc0"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/jammy-server-cloudimg-amd64-disk-kvm.img","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/astoria-test-1.img","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/astoria-test-1-cidata.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.5","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk1"}' \
-netdev '{"type":"tap","fd":"35","vhost":true,"vhostfd":"37","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:ba:09:03","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=33,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.7","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
libnuma: Warning: /sys not mounted or invalid. Assuming one node: Permission denied
2023-11-16T13:43:54.873250Z qemu-system-x86_64: The nodes select no CPUs
2023-11-16 13:43:54.988+0000: shutting down, reason=failed
2023-11-16 13:37:10.993+0000: starting up libvirt version: 9.0.0, package: 9.0.0-4 (Debian), qemu version: 7.2.5Debian 1:7.2+dfsg-7+deb12u2, kernel: 6.1.0-13-amd64, hostname: astoria.[snip]
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-astoria-test-1/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=astoria-test-1,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-astoria-test-1/master-key.aes"}' \
-machine pc-q35-7.2,usb=off,dump-guest-core=off \
-accel kvm \
-cpu host,migratable=on \
-m size=4194304k,slots=1,maxmem=4210688k \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-object '{"qom-type":"memory-backend-ram","id":"ram-node0","size":4294967296,"host-nodes":[0],"policy":"bind"}' \
-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-uuid e74129a0-2e80-45af-88d4-e6949d7dd628 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"pcie-root-port","port":15,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x7"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/jammy-server-cloudimg-amd64-disk-kvm.img","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/astoria-test-1.img","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/astoria-test-1-cidata.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.5","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk1"}' \
-netdev '{"type":"tap","fd":"35","vhost":true,"vhostfd":"37","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:ba:09:03","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=33,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.7","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/4 (label charserial0)
2023-11-16T13:38:39.516358Z qemu-system-x86_64: terminating on signal 15 from pid 95295 (/usr/sbin/libvirtd)
2023-11-16 13:38:39.804+0000: shutting down, reason=shutdown
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ Users mailing list -- users@xxxxxxxxxxxxxxxxx To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxx
