On 4/7/23 15:12, Andrea Bolognani wrote:
On Tue, Mar 28, 2023 at 04:46:45PM -0300, Daniel Henrique Barboza wrote:
On 3/22/23 13:37, Andrea Bolognani wrote:
On Wed, Mar 22, 2023 at 10:36:20AM -0300, Daniel Henrique Barboza wrote:
I'm not sure if the OS overwrites the firmware when running bare metal. Usually
they provide different OS images for QEMU/libvirt and bare metal systems, probably
to account for these differences. Baking the firmware in the kernel like Fedora
Rawhide is doing was important a few years ago when RISC-V QEMU wasn't loading
the firmware by default, but now it's getting in the way.
All this said, having a look at the recently updated Fedora for RISC-V wiki, the
instructions for booting with libvirt and QEMU are different. libvirt [1] is using
the '-bios none' attribute with 'virt-install' + a Fedora Rawhide image, but
QEMU instructions [2] doesn't use '-bios none' and it's using Fedora 37.
The libvirt instructions in that page seem to have been updated in
the Fedora 30 timeframe, whereas the QEMU instructions appear to be
current as of Fedora 37.
At first I don't see any particular reason of why this Fedora 37 image would work
on QEMU and not on libvirt. So what t I'll do now is do some testings with libvirt
and Fedora 37. If it works then this series becomes way less important.
The libvirt instructions tell you to use
--boot kernel=Fedora-Developer-Rawhide-*-fw_payload-uboot-qemu-virt-smode.elf
--qemu-commandline='-bios none'
(aka -bios none -kernel foo) while the QEMU ones suggest
-bios u/usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin
-device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000
Those are completely different approaches to booting the guest OS.
The latter is much saner IMO, because it keeps the firmware under
control of the host (as is the case for SeaBIOS/edk2) instead of
mixing the kernel and the firmware and requiring guest-specific files
to be extracted from the disk image before being able to boot.
libvirt already provides the ability to pass arbitrary paths to -bios
via the <loader type='rom'> element, so making the first part work
should just be a matter of pointing it to the correct path. We don't
seem to have the -device loader part wired up though, so that would
need to be added. Probably as an additional attribute, in the vein of
nvram.template? The address would have to be specified as well.
Note that the firmware descriptor format already supports u-boot as
the firmware type. So in the long run ideally you'd only need to
specify
<os firmware='uboot'>
and, assuming the uboot-images-riscv64 package is installed on the
host, everything should just work.
So, I got the chance to try this out. We can remove the '-loader' from the
Fedora 37 command line. This:
-device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000
Is loading the u-boot image in the address that the 'virt' machine loads the kernel.
So in the end it's similar to:
-kernel u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb
Excellent insight! I've confirmed that you are correct, and that
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000'/>
</qemu:commandline>
can indeed be replaced with
<kernel>u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb</kernel>
without apparently affecting the VM's ability to boot.
This means that RISC-V Fedora 37 is bootable without any libvirt changes. Similar to
what we already do with Ubuntu.
I've spent some time on this as well. Here are my findings.
First, let's briefly mention how RISC-V VMs had to be booted a few
years back. This is described in detail at [1] and preceding
sections, but the quick version is that you would have a single file
containing (I believe) OpenSBI, u-boot and the kernel. This file
would need to be extracted from the disk image, and would then be
passed to -kernel. Having the firmware and the kernel in a single
file, which lives on the host rather than the guest, was obviously
extremely cumbersome and unsuitable for scenarios beyond
experimentation.
Today, things are looking a lot better: the firmware lives in the
host and the kernel lives in the guest, as they should. However,
there seem to be two different approaches to loading the firmware
floating around.
The first one, described in the Fedora wiki[2], involves passing two
separate files to QEMU. They both come from the uboot-images-riscv64
package, and their use looks like
-bios /usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin
-kernel /usr/share/uboot/qemu-riscv64_spl/u-boot.itb
As far as I understand, this is a u-boot binary, just divided in two
parts and loaded separately. They might run in different hardware
privilege contexts? I think the first file might also embed OpenSBI
somehow.
The second approach is the one described in the Ubuntu wiki[3], and
also requires passing two files to QEMU, except this time they come
from the opensbi and u-boot-qemu packages respectively. The usage
looks like
-bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf
-kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf
I think in this case the first file is a minimal build of OpenSBI
that likely just initializes enough hardware before handing control
to an arbitrary payload - in this case, u-boot.
There's a twist, however: the Ubuntu wiki claims that the -bios part
is not necessary with QEMU >= 7.0, and I've verified that at least
with the QEMU 7.2 package in Fedora 37 this is accurate. I believe
this is because opensbi-riscv64-generic-fw_dynamic.bin, which is
included in the QEMU package, gets loaded automatically, and this
build of OpenSBI apparently also knows how to jump ahead to where
-kernel loads u-boot.
Yeah, until not so long ago QEMU wasn't loading any default OpenSBI firmware,
requiring the distro/user to manually add a -bios entry. Hopefully distros
can now use the default QEMU OpenSBI binary.
I've tried mixing things, that is, boot the Fedora VM using the
firmware taken from Ubuntu and vice versa, and thankfully it looks
like this is perfectly fine.
In conclusion, the current version of libvirt is perfectly capable of
booting modern RISC-V guests through a combination of <loader> and
<kernel> elements.
That said, even if things nominally work, way too much handholding is
necessary. So I'm wondering how we can push things forward to make
the process more user-friendly.
The obvious first step would be for the u-boot-qemu package in Ubuntu
to ship a JSON descriptor file containing something like
{
"interface-types": [
"u-boot"
],
"mapping": {
"device": "kernel",
"filename": "/usr/lib/u-boot/qemu-riscv64_smode/uboot.elf"
},
"targets": [
{
"architecture": "riscv64",
"machines": [
"virt"
]
}
]
}
So, I talked with Canonical a few months back about this package. I'm using a F37
dev box and I had to uncompress the u-boot .deb package by hand to extract the
uboot.elf file to be able to launch the guest, meaning that users that aren't using
an Ubuntu host would need to go through that to boot an Ubuntu RISC-V domain.
I got told that any u-boot build would work. Didn't have the opportunity to test
it.
I also believe that there's nothing particularly different in how RISC-V works to
require a -kernel argument every time. Users don't have to do that with x86 and
even ppc64 pSeries domains: you give a disk with the guest image and everything
just works. Ideally we would want that for RISC-V domains as well. We'll probably
need stuff done in QEMU and OpenSBI to support this use case but I believe it's
worth it.
Depending on how much work/how long it would take we can discuss how libvirt can
ease the users pain in the meantime.
With some relatively straightforward plumbing on the libvirt side, we
should be able to use the information provided in that file to make
the simple VM configuration
<os firmware='u-boot'>
work out of the box.
Getting the Fedora version to work would be trickier. As far as I can
tell, there's currently no spec-compliant way to describe a firmware
that requires both -bios and -kernel to be used at the same time.
Should the spec be extended? Can we get Fedora to standardize on the,
at least to an outside observer, simpler approach that Ubuntu has
adopted?
I believe this has to be answered by the Fedora folks, but yeah, it would be nice
if Fedora could adopt the same approach that Ubuntu, Debian, OpenSuse and others
are using.
Thanks,
Daniel
[1] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_with_libvirt
[2] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_under_QEMU
[3] https://wiki.ubuntu.com/RISC-V#Booting_with_QEMU
