----- Forwarded message from "Richard W.M. Jones" <rjones@xxxxxxxxxx> ----- Date: Thu, 4 Jan 2018 11:29:51 +0000 From: "Richard W.M. Jones" <rjones@xxxxxxxxxx> To: sw-dev@xxxxxxxxxxxxxxxx Subject: [sw-dev] Status of the second bootstrap of Fedora/RISC-V I've almost reached the end of the allotted time available for bootstrapping Fedora/RISC-V for a second time, so this is a status report describing what I found and how far I got. Background ---------- Fedora is a binary Linux distribution. I first bootstrapped Fedora on RISC-V at the end of 2016 (the "first bootstrap"). https://fedoraproject.org/wiki/Architectures/RISC-V To do a Linux distro sanely we need full ABI guarantees at least between userspace and the kernel, so that involves mainly glibc and the kernel, and at that time the ABI was changing which meant we would need to go through the very costly process of re-bootstrapping everything multiple times. So after porting a large percentage (I think about 1/3rd) of all Fedora packages to the (then-) old glibc, I stopped work on it for about a year. glibc is supposed to go upstream in a few months from now, and that will guarantee a stable ABI between userspace and the kernel, allowing us to sanely build a Linux distro. I therefore put aside some time now to practice bootstrapping Fedora again (the "second bootstrap"). When glibc finally goes upstream we will need to do the third and final bootstrap, and from that point on older Fedora/RISC-V releases will be used to build each new Fedora release. Bootstrapping stages -------------------- [These are historically named and based on the stages we used for aarch64.] stages 1 & 2: QEMU and cross-compiler are built from riscv-qemu and riscv-tools. https://github.com/rwmjones/fedora-riscv-bootstrap/blob/1858bd496378ddcce88a63c5ceda5483d7b4fdbe/Makefile#L21 https://github.com/rwmjones/fedora-riscv-bootstrap/blob/1858bd496378ddcce88a63c5ceda5483d7b4fdbe/Makefile#L71 stage 3: We build a minimal Fedora/x86_64 chroot and remove all x86_64 ELF binaries and libraries. Using the hosted cross-compiler we build RISC-V binaries and libraries and to replace the x86 ones. We then build a disk image from the chroot (it has many other hacky aspects to it) and boot it under qemu. https://github.com/rwmjones/fedora-riscv-bootstrap/blob/1858bd496378ddcce88a63c5ceda5483d7b4fdbe/Makefile#L117 The stage 3 disk image is just enough to run ‘rpmbuild’ and ‘gcc’ and a small handful of other tools, which is just enough to build RPMs. https://github.com/rwmjones/fedora-riscv-bootstrap/blob/1858bd496378ddcce88a63c5ceda5483d7b4fdbe/Makefile#L1386 stage 4: Using only RPMs generated from stage 3 we build a pristine disk image. This disk image contains only files controlled by RPMs [actually there are two additional files needed: /init and a poweroff binary, both eventually will be replaced by systemd]. https://github.com/rwmjones/fedora-riscv-bootstrap/blob/1858bd496378ddcce88a63c5ceda5483d7b4fdbe/Makefile#L1420 Using the stage 4 disk image we then build the rest of the Fedora packages. This requires some manual intervention, usually to break circular chains of dependencies of which there are many. There is also an autobuilder which can build packages from Fedora alphabetically or by shadowing the Fedora Koji build system. The autobuilder will need rewriting at some point since it can be made much more efficient now that we have working networking. Status of the second bootstrap ------------------------------ I spent about 10 working days on this, and got a large part of the way through stage 3. You can see the status and download built packages here: https://github.com/rwmjones/fedora-riscv-bootstrap There is also a stage3 disk image here: http://oirase.annexia.org/riscv/ The eagle-eyed will notice that I'm still building some Fedora 24/25 packages (latest is Fedora 27). This is because those packages contain all the fixes from the first time around so it's convenient to use them for the moment. Even with these packages it should be sufficient to build Fedora 27 in stage 4 (particularly as packages get replaced with the new versions as we go along). Unfortunately I did not yet get a working stage 4 disk image. A glibc RPM is required for stage 4 since almost all packages depend on it, but the glibc build is hanging at some point for unclear reasons. Debugging anything inside the stage 3 QEMU instance is a recipe for pain and also debugging tools don't work inside stage 3. QEMU user networking (with virtio-net virtual device) worked fine for me, but I wasn't able to compile enough dependencies to get dhcp to work. Problems -------- There is definitely a problem with GCC miscompiling with -O2 (which can be worked around using -O0). It affected at least 4 packages, but I was not able to produce any sort of minimal test case or common cause. The issue is being tracked in: https://github.com/riscv/riscv-gcc/issues/100 There is some bug in the kernel which causes it to hit a BUG_ON in fs/buffer.c. Latest status is: https://groups.google.com/a/groups.riscv.org/d/msg/sw-dev/v05FjcGC1EI/atXXUAcsCgAJ Poweroff does not work, however I was able to work around it: https://groups.google.com/a/groups.riscv.org/forum/#!topic/sw-dev/8ps3VtdXpBk The following bug affects iputils. I didn't try the upstream fix yet: https://github.com/riscv/riscv-binutils-gdb/issues/108 A notable problem was building python2 which has multiple build issues (although python3 was fine). Luckily Fedora has greatly reduced its dependence on python2 and no important system packages should depend on it in F27. In fact for the third bootstrap I may not even bother with python2. I wasn't able to get SMP to work, but unless mttcg also works SMP wouldn't be helpful anyway. Future ------ As discussed in the first section, my plan now is to wait until the glibc changes are accepted upstream and released (hopefully in glibc 2.27). At that point I will go for the third and final bootstrap. Rich. ----- End forwarded message ----- -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
